OGSA-DAI Activities
Amy Krause
a.krause@epcc.ed.ac.uk
Goals for OGSA-DAI

Aim to deliver application mechanisms that:

Meet the data requirements of Grid applications




Acceptable and supportable by database providers



Functionality, performance and reliability
Reduce development cost of data centric Grid applications
Provide consistent interfaces to data resources
Trustable, imposed demand is acceptable, etc.
Provide a standard framework that satisfies standard
requirements
A base for developing higher-level services




Data federation
Distributed query processing
Data mining
Data visualisation
UK Globus Week - www.ogsadai.org.uk
2
Core features of OGSA-DAI

An extensible framework for building applications

Supports relational, xml and some files


Supports various delivery options





XSLT, ZIP, GZip
Supports message level security using X509
Client Toolkit library for application developers
Comprehensive documentation and tutorials
Fourth production release coming up:


SOAP, FTP, GridFTP, HTTP, files, email, inter-service
Supports various transforms


MySQL, Oracle, DB2, SQL Server, Postgres, XIndice, CSV, EMBL
OGSI (GT3), WS-I (OMII) and WS-RF (GT4) flavours
Highly extensible

Strength is in customising out-of-box features
UK Globus Week - www.ogsadai.org.uk
3
Data Service Internals
element
Query
Activity
query
response
document
The
Engine
perform
document
data
element
element
Transform
Activity
data
Delivery
Activity
credentials
data
connection
connection
Connection
Manager
role
UK Globus Week - www.ogsadai.org.uk
Role Mapper
4
The Activity Engine


Engine is the central OGSA-DAI component
Dictates behaviour when perform documents are
submitted

Parses and validates perform document

Identifies required activities implementations

Processes activities

Composes response document

Returns response document to the service layer
perform
document
The
Engine
UK Globus Week - www.ogsadai.org.uk
response
document
5
Perform Documents

Perform documents

Encapsulate a serialisation of multiple interactions with a
service into a single interaction

Abstract each interaction into an “activity”

Data can flow from one activity to another

No control constructs present


no conditionals, loops or variables
Not intended for human consumption

Generated and processed by client toolkit
Perform
Query
Transform
Deliver
UK Globus Week - www.ogsadai.org.uk
6
Activities

An Activity dictates an action to be performed

Query a data resource

Transform data

Deliver results

Engine processes a sequence of activities

Subset of activities available to a Data Resource


Specified in configuration files
Data can flow between activities
SQL
Query
Statement
WebRowSet
xml data
XSLT
Transform
HTML
data
UK Globus Week - www.ogsadai.org.uk
Delivery
ToURL
7
Activity Taxonomy

Activities fall into three main functional groups
Activity
Statement

Interact with the data resource, e.g. direct an SQL query to a
DBMS
Delivery


Transform
Statement


Delivery
Deliver data to a third party
Transform

Perform transformations on data, e.g. XSL Transform,
compression
UK Globus Week - www.ogsadai.org.uk
8
Building Blocks – Predefined
Activities
DeliverFromGDT
xmlCollectionManagement
relationalResourceManager
xmlResourceManagement
sqlBulkLoadRowset
sqlUpdateStatement
sqlStoredProcedure
sqlQueryStatement
xQueryStatement
xUpdateStatement
xPathStatement
DeliverToGDT
DeliverToStream
outputStream
DeliverFromGFTP
DeliverToGFTP
DeliverToURL
DeliverFromURL
UK Globus Week - www.ogsadai.org.uk
inputStream
xslTransform
zipArchive
gzipCompression
9
Activity Implementations


Extensibility point
All Activity implementations extend the
abstract Activity class
Activity
# mContext: Context
# mElement: Element
# mInputs: String[]
# mOutputs: String[]
+ Activity( element: Element )
+ processBlock() : void
+ setContext( context: Context ) : void
+ getStatus() : int
# setStatus( status: int ) : void
UK Globus Week - www.ogsadai.org.uk
10
Custom Activities

Users can develop additional activities

To support different query languages


To perform different kinds of transformation


e.g. STX
To deliver results using a different mechanism


e.g. XQuery
e.g. WebDAV
An activity requires
XSD schema sql_query_statement.xsd
 Java implementation
SQLQueryStatementActivity.java

UK Globus Week - www.ogsadai.org.uk
11
Activity Inputs and Outputs


Activities read and write blocks of data

Allows efficient streaming between activities

Reduces memory overhead
A block is a Java Object


Untyped but usually a String or byte array
Interfaces for reading and writing

BlockReader and BlockWriter
SQL
Query
Statement
XSL
Transform
Activity
UK Globus Week - www.ogsadai.org.uk
Deliver
To URL
12
Connection Managers


Govern access to a data resource

Open/close connections

Validate user credentials using a RoleMapper

Facilitate connection pooling
Provided for JDBC, XML:DB and File Systems
open connection
get connection
JDBC
Connection
Manager
Relational
database
close connection
return connection
UK Globus Week - www.ogsadai.org.uk
SQL
Query
Statement
13
Sequence Diagram
:Activity
:Context
:DataResource
Implementation
:RoleMapper
:DatabaseRole
Get user credentials and data resource implementation
Get connection using user credentials
Get database role using user credentials
Get user ID and password
open connection using
user ID and password
Do exciting things with the connection
Return connection
UK Globus Week - www.ogsadai.org.uk
14
Advantages of the Activity Model

Avoid multiple message exchanges

Extensible



Developers can add functionality
Could import third party trusted
activities
Allows for optimisation

Engine can optimise internals
UK Globus Week - www.ogsadai.org.uk
15
Summary

The Engine is the central component of a data service

Activities perform actions



Querying, Updating

Transforming

Delivering
Data Resource Implementations manage access to
underlying data resources
Architecture designed for extensibility

New Activities

New Role Mappers

New Connection Managers
UK Globus Week - www.ogsadai.org.uk
17
OGSA-DAI Client Toolkit
Amy Krause
a.krause@epcc.ed.ac.uk
Overview


Client Toolkit will be released in OGSI
(GT3), WS-I (OMII) and WS-RF (GT4)
flavours
Common abstraction level for all flavours


R6: Clients must know which version the
service supports
Planned for R7: Service version is hidden
from the user
UK Globus Week - www.ogsadai.org.uk
19
Client Toolkit
ServiceFetcher
getDataService(URL)
getRegistry(URL)
Registry
getServices()
Client
perform(doc)
DataService
UK Globus Week - www.ogsadai.org.uk
20
Client Toolkit for WSRF
WSRF
ServiceFetcher
getDataService(URL)
getRegistry(URL)
WSRF
Registry
getServices()
WSRF
Client
perform(doc)
WSRF
DataService
UK Globus Week - www.ogsadai.org.uk
21
Service Fetcher

WSRFServiceFetcher

getDataService(URL);

getRegistry(URL);
UK Globus Week - www.ogsadai.org.uk
22
<<interface>>
Service
<<interface>>
DataTransport
<<interface>>
Properties
<<interface>>
DAIService
<<interface>>
DataService
OGSI
DataService
WSRF
DataService
UK Globus Week - www.ogsadai.org.uk
WSI
DataService
23
Data Services

DataService

getResources();

getPropertyNames();

getProperty(QName);

perform(Request);

perform(Activity);

putBlock(Data);

getBlock();

…
UK Globus Week - www.ogsadai.org.uk
24
Interaction with a Data Service

Client sends a request to a data service

A request contains a set of activities
Client
Activity
Data Service
Activity
Activity
Request
UK Globus Week - www.ogsadai.org.uk
25
Interaction with a Data Service


The Data service processes the request
Returns a response document with a result for each
activity
Client
Result
Data Service
Result
Result
Response
UK Globus Week - www.ogsadai.org.uk
26
Activities and Requests

A request contains a set of activities

An activity dictates an action to be performed


Query a data resource

Transform data

Deliver results
Data can flow between activities
SQL
Query
Statement
web rowset
data
XSLT
Transform
HTML
data
UK Globus Week - www.ogsadai.org.uk
Deliver
ToURL
27
Activity Implementations



Activity implementation are shared among
the different flavours of the client toolkit
Makes it easy to translate an existing client
from one flavour to another
Properties are represented in the same
way

Although published properties may differ
between data services
UK Globus Week - www.ogsadai.org.uk
28
Delivery Methods
GridFTP server
DeliverTo/FromGFTP
Web Server
DeliverFromURL
Local
Filesystem
DeliverTo/FromFile
Data Service
FTP
server
DeliverTo/FromURL
UK Globus Week - www.ogsadai.org.uk
29
Delivering data to another GDS



The GDT port type allows to transfer data
from one data service to another.
An InputStream activity of GDS1 connects
to a DeliverToDT activity of GDS2
Alternatively, an OutputStream activity can
be connected to a DeliverFromDT activity
DS1
DeliverToDT
InputStream
UK Globus Week - www.ogsadai.org.uk
DS2
30
Data Integration Scenario
MySQL
database
DS2
deliver
DS3
deliver
select +
deliverToGDT
Oracle
database
inputStream
from DS1, DS2
join tables
DB2
database
DS1
select +
deliverToGDT
Client
UK Globus Week - www.ogsadai.org.uk
31
Select + Deliver

Get a service object from the ServiceFetcher:
DataService service1 =
WSRFServiceFetcher
.getDataService(“http://...”);
UK Globus Week - www.ogsadai.org.uk
32
Select + Deliver

Construct a new request:
ActivityRequest request =
new ActivityRequest();

Set up the SQL query and delivery instructions:
SQLQuery query = new SQLQuery(
“select * from table where …”);
DeliverToDT deliver = new DeliverToDT(
“http://host/services/DataService”);
UK Globus Week - www.ogsadai.org.uk
33
Select + Deliver

Connect input and outputs:
deliver.setInput(query.getOutput());

Register activities with the request:
request.add(query);
request.add(deliver);

Perform the request:
service1.perform(request);
UK Globus Week - www.ogsadai.org.uk
34
Data Integration Scenario
MySQL
database
DS2
DS3
Oracle
database
select +
deliverToGDT
DB2
database
DS1
select +
deliverToGDT
Client
UK Globus Week - www.ogsadai.org.uk
35
Data Integration Scenario
MySQL
database
DS2
deliver
DS3
Oracle
database
deliver
DB2
database
DS1
Client
UK Globus Week - www.ogsadai.org.uk
36
Input Stream and Bulk Load

Set up the input stream that listens for incoming
data and bulk load the data into the database
InputStream input = new InputStream(“inp1”);
SQLBulkLoad bulkload =
new SQLBulkLoad(“table”);
bulkload.setInput(input.getOutput());
request.add(input);
request.add(bulkload);
service3.perform(request);
UK Globus Week - www.ogsadai.org.uk
37
Data Integration Scenario
MySQL
database
DS2
DS3
Oracle
database
inputStream
from DS1, DS2
DB2
database
DS1
Client
UK Globus Week - www.ogsadai.org.uk
38
Further information

The OGSA-DAI Project Site:


The DAIS-WG site:



users@ogsadai.org.uk
General discussion on grid DAI matters
Formal support for OGSA-DAI releases



http://forge.gridforum.org/projects/dais-wg/
OGSA-DAI Users Mailing list


http://www.ogsadai.org.uk
http://www.ogsadai.org.uk/support
support@ogsadai.org.uk
OGSA-DAI training courses
UK Globus Week - www.ogsadai.org.uk
39