OGSA-DAI Users’
Meeting
Introduction
Malcolm Atkinson
Director
www.nesc.ac.uk
7th April 2004
Overview
Today’s Meeting
Users meet each other
We hear Users’ Complaints, Requirements & Priorities
Users understand better the limits of one team’s work
We jointly identify the way ahead
OGSA-DAI Vision
What we aspire to achieve
Short, medium and long term
Relationship with GGF DAIS WG Standardisation effort
OGSA-DAI Today
Current and Imminent Releases
The Road Map
Which you — the Users — should influence
OGSA-DAI History Lesson
DB & BiGUM meetings
August 2001 — Paul Watson paper — prime mover
Database Task Force
01Q3 – 02Q3


Norman Paton, Dave Pearson & Paul Watson
http://www.cs.man.ac.uk/grid-db/dbtf.html
Architecture Task Force
01Q3 – 02Q2


Urged UK engagement in OGSA
OGSI Sep 01 to Sep 03 http://www.nesc.ac.uk/teams/atf/01-03/
OGSA-DAI project
Start Feb 02 – end Sept 03
Produced releases R1 to R3.1
Produced Alpha Release of DQP
http://www.ogsadai.org.uk/
DAIT project
Start Nov 03 – end ~ March 06
Continues OGSA-DAI development
• GGF4 Toronto March 02
• GGF5
July 02
•DAISEdinburgh
BoF
•DAIS
launched
•OGSI WG
announced
• GGF10 Berlin March 04
•WS-Resource Framework
OGSA-DAI Partners
IBM
USA
EPCC & NeSC
Glasgow
Newcastle
Belfast
Daresbury Lab
Manchester
Oxford
Cambridge
Oracle
RAL
Cardiff
London
IBM Hursley
Southampton
Hinxton
$5 million, 20 months, started February 2002
Additional 24 months, starts October 2003
OGSA-DAI Vision 1
There are many sources of Data
Structure is the essence of using data
Structure supported by DBMS: Relational & XML
Structure also recognised in files: BinX & GGF DFDL
Sources of Data are Autonomously Managed
Vary in model, structure, schema
Vary in content
Vary in Access & Use policies
Vary in Platform
Discovery & Design enabled by Combining Data
From multiple sources
Recorded in multiple data resources
OGSA-DAI Vision 2
Developing applications using heterogeneous data
sources is difficult:
Dealing with all those variations
Negotiating with data owners and data platforms
Communication protocols and data movement
Data translation
Distributed operations
Scale, reliability, coping with change, failure recovery, security,
privacy
…
Currently solved incrementally on a per application basis
OGSA-DAI to the rescue
OGSA-DAI Vision 3
OGSA-DAI will provide a generic framework
Standard components for dealing with common requirements

Query, update, bulk load, schema edit, metadata access, transactions, …
Agreed protocols and standard agreements with DBMS
Established mechanisms for handling owner policy requirements
Provides data translation
Supports distributed operations
Delivers

Scale, reliability, coping with change, failure recovery, security, privacy
…
Costs of developing a robust OGSA-DAI amortised over many
applications
Development of Applications significantly accelerated
Accepted de facto standard by DBMS vendors and data owners
Efficiency from direct support and system or resource knowledge
Collaborating International Community Extending the Framework
E.G. the work of OGSA-WebDB
OGSA-DAI Vision 4
Progressively develop the functionality
Progressively raise level of operations
Progressively improve quality
Scale, Performance, Reliability, Flexibility, Usability
Adapt to changing context
Support user community
Training, Documentation, Consultation, Collaboration
Place High Value on Users’ & Application Developers’
Ingenuity
They know about the data resources
They invent new ways of combining and analysing data
Therefore Extensibility and Flexibility MUST always be
supported
OGSA-DAI today
1919 Downloads so far from http://www.ogsadai.org.uk/
Release 4: April 04 from http://www.ogsadai.org.uk/
Release 5: October 04 from http://www.ogsadai.org.uk/
Releases 6&7: 2005: Many options — You must help us choose
Download analysis
Downloads by Domain
gov, 13
at, 44
it, 12
br, 11
ru, 9
net, 73
de, 87
unresolved, 674
jp, 126
edu, 133
com, 170
uk, 487
And Now …
Users’ Presentations
What are you doing with our “baby”
What do you like
What do you want changed, extended or added
What consensus is there
Team Presentations
More details of current and planned release
The issues that face us


Choosing Interoperation Platforms
Choosing which requirements get priority
Working Groups
Collation of the Groups’ conclusions
Ideas for Future User Group Activities
URGENT
Questions
?
Reserve Slides Follow
OGSA-DAI Behaviours
Request to Registry for sources of
data about “x”
Registry responds
with Factory
handle
Analyst
SOAP/HTTP
Registry
GDSR
service creation
API interactions
Request to Factory for access to
database
Factory returns handle of
GDS to client
Factory
GDSF
Factory creates
GridDataService
Client queries GDS with SQL,
XPath, XQuery etc
Query results
returned XML
OR
delivered to consumer
as XML
Consumer
Grid Data
Service
GDS
Database
(Xindice, MySQL
Oracle, DB2)
GDS interacts
with database
Delivery Patterns
Retrieve
1.
Update/Insert
4.
Q
A
Q+U
7.
A
G
Pipeline
Q1
G1 = P
G
S
S+R
S1
A
2.
G
A
S
6.
Q
A
C
Q1 + D
P
U
S1
A
Q
U/R
Q2
I
A
D
R
G1 = P
8.
G
S
G2 = C
G
R
C
S2
Q+D
S
I
I
P
A
3.
U
5.
Q+D
U/R
Q2 + D
G
S
S2
G2 = C
Statement Operations
Receives statement
Performs statement against a data resource
Returns results
Statement
Statement Component and
Database Connections
Data Resource
Results
Statement Document
<?xml version="1.0" encoding="UTF-8" ?>
<GridDataServiceScript>
<Header> … Name, versioning, configuration specification </Header>
<Body>
<Documentation>Update with data delivered with the script</Documentation>
<DefineParameter name="table">myimages</DefineParameter>
<DefineParameter name="id">id = 2</DefineParameter>
<Statement name="xyz" dataResource="MyDataResource">
SELECT * FROM <UseParameter reference="table" />
WHERE <UseParameter reference="id" />
</Statement>
<Delivery name="delivery1">
<Mechanism type="bulk" />
<Mode type="full" />
<From>xyz</From>
<To>response</To>
</Delivery>
<Execute name="execute1">xyz</Execute>
</Body>
</GridDataServiceScript>
Statement Architecture
Engine/GDS-S
Statement
Statement Activity
Statement Handlers
JDBC Statement Handlers
XMLDB Statement Handlers
Relational
DB
JDBC Connection
XMLDB Connection
JDBC API
XML:DB API
XML DB
OGSA-DAI Security
Grid Security Infrastructure Authentication
The role mapper takes credentials and a database name and
returns a dbRole
dbRole contains username and password
Credentials
Database Name
Role
Mapper
username
dbRole
password
Data Access & Integration Services
1a. Request to Registry
for sources of data
about “x”
SOAP/HTTP
Registry
1b. Registry
responds with
Factory handle
service creation
API interactions
2a. Request to Factory for access
to database
Factory
Client
2c. Factory returns
handle of GDS to
client
3a. Client queries GDS with
XPath, SQL, etc
3c. Results of query returned to
client as XML
2b. Factory creates
GridDataService to manage
access
Grid Data
Service
XML /
Relationa
l
database
3b. GDS interacts with database
Future DAI Services?
1a. Request to Registry for
sources of data about “x” &
“y”
1b. Registry
responds with
Factory handle
Data
Registry
SOAP/HTTP
service creation
API interactions
2a. Request to Factory for access and
integration from resources Sx and Sy
Data Access
& Integration
master
2c. Factory
returns handle of GDS to client
3b.
Client
Problem
tells“scientific”
Solving
analyst
Client
Application
Environment
coding
scientific
insights
Analyst
2b. Factory creates
Semantic
GridDataServices network
Meta data
3a. Client submits sequence of
scripts each has a set of queries
to GDS with XPath, SQL, etc
GDTS1
GDS
GDTS
XML
database
GDS2
Sx
3c. Sequences of result sets returned to
analyst as formatted binary described in
a standard XML notation
Application Code
GDS
GDS1
Sy
GDS3
GDS
GDTS2
GDTS
Relational
database