gUSE Services
Remote API, DCI Bridge, Data Bridge, Robot Certificate
Zoltán Farkas, Péter Kacsuk, Gábor Herman,
István Márton, Tibor Gottdank, Ákos Balaskó
MTA SZTAKI LPDS
zoltan.farkas@sztaki.mta.hu
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI-283481
Outline
•
•
•
•
Remote API
DCI Bridge
Data Bridge
Robot Certificate
REMOTE API
Goals - Solutions
•
•
•
•
•
Submitting a WS-PGRADE/gUSE workflow using an HTTP client without the web
interface of WS-PGRADE
Solution: The Remote API web service extension of gUSE will be used as
application layer in order to communicate with the backend (gUSE) instead of WSPGRADE portlets
The workflows to be submit should be available at the caller
– As WS-PGRADE workflow definitions contain own set of input files, reference
to input files must be changed, but the files of job executable, and the
command line parameters can be changed as well
Solution: Manual change of files to be replaced and the creation of new
association table (coded in the „portmapping.txt” file). The main workflow
descriptor file <workflow>.xml should be modified only in the case when
command line parameters belonging to the individual nodes (jobs) of the
workflow must be modified
On completion of a workflow submission session the common resources should
be cleared
Solution: The servers side data belonging to the workflow (submission) will be
cleared upon the successful execution of the „Get output” command of the client
Typical workflow development scenario in case of remote call
HTTP Client
GRAPH Ed
Upd. user files
<wf>.zip
<wf>.xml
Browser
Copy (modify)
tree of
<wf> files
portmapping.txt
Copy and rename
Orig user files
1.Create
and
submit
original
workflow
2. Download
tested
workflow
<wf>.xml
3. Reengineering
of wf definition
<inputs>.zip
<file_i>
<file_i>
<file_i>
Eventual update
4. Submit
the
modified
workflow
Client side
Server side
Portal
Portal
Remote API Serv.
WS PGRADE SERVER 2
WS PGRADE SERVER 1
gUSE
gUSE
DCI Bridge
DCI Bridge
Set of
computational
resources
Summary of workflow development
and remote submission process
1.
2.
Download the original workflow from WS-PGRADE
Do some reengineering:
•
•
•
3.
4.
separate the structure description from the other parts
explore the user defined (not channel) input files connected to the
numbered ports of the named jobs and the files of executable of the
named jobs
Pack the needed files together and create a text file which describes the
association of the files and the named jobs and ports
Create the needed script files needed to habitual submission
process (call, observe, download (delete))
Execute the scripts on the client machine
Basic assumptions of remote call setup
• There is a working gUSE set of services where the Remote API Servlet
extension has been installed
• There is a client machine containing the description (structure, input files)
of a workflow
• The client can reach the server by the HTTP protocol
• There is a server-created password known by the client(s)
• The client has the necessary proxy certificate file if the submissions
involved in the workflow are directed to such resources which need
certificate bound authorization
*(see installation file and documentation on Sourceforge for version <V> :
Code:
http://sourceforge.net/projects/guse/files/<V>/remote-3.4.tgz/download
Documentation:
http://sourceforge.net/projects/guse/files/<V>/Documentation/RemoteAPI_Install_Manual.pdf/do
wnload
)
DCI Bridge
What is DCI Bridge - 1
Supported DCIs
A web service based application that provides standard access to various distributed
computing infrastructures (DCIs), such as: grids, desktop grids, clusters, clouds and
service based computational resources (connecting through its DCI plugins to the
external DCI resources).
What is DCI Bridge - 2
Generic View
When a user submits a workflow, its job components can be submitted transparently into
the various DCI systems using the OGSA Basic Execution Service 1.0 (BES) interface. As a
result, the access protocol and all the technical details of the various DCI systems are totally
hidden behind the BES interface. The standardized job description language of BES is JSDL.
From gUSE or independently
(from other workflow systems)
can also use DCI Bridge for job
submission
A sample view of
Middleware settings
page.
http://LOCLAHOST:8080/dci_bridge_service/conf
Administration
Interface – 1
Base Menus
Middleware Menus
Administration
Interface – 2
Submenu name
Manager
Settings
Log entries
Functionality
The general flow of jobs can be enabled or disabled in the Manager submenu.
It contains properties for generic settings.
Opens the history log where the main user activities of the Administrator can be
traced.
Submenu name
Add new
Edit
Monitor
Middleware settings
Log entries
Functionality
A new resource reference can be named and added to the group
An existing resource can be selected and its access attributes can be modified
Observation of jobs belonging to the given resource group
Set of generic parameters common in the given middleware
Opens the history log where the main user activities of the Administrator can be
traced.
The job flow in the DCI Bridge and between
the components.
Architecture
WFI generates and submits the jobs'
JSDL to the DCI-BRIDGE.
plugins
Plugins
WFI
WS-PGRADE
JSDL Processing
JSDL
<ns3:SDL_Type>
<ns3:Constraints>
<ns3:Middleware>
<ns3:DCIName>local</ns3:DCIName>
<ns3:MyProxy/>
<ns3:ManagedResource>dci-bridge host(64bit)</ns3:ManagedResource>
</ns3:Middleware>
<ns3:Budget>0</ns3:Budget>
</ns3:Constraints>
</ns3:SDL_Type>
Admin Interface (JSP pages)
DCI Bridge
BES
Semi-autogenerated Eventhandlers
Middleware Brokering and Management Layer (Plugin Manager)
Middleware Class
Unicore
gLite
GT2
GT4
GT5
Local
...
PBS
CB
DCI Bridge accepts standardized JSDL job description documents. These documents
are based on a well-defined XML scheme containing information about the job inputs,
binaries, runtime settings and output locations.
DCI Bridge
Job Processing
2
WFI sends JSDL to DCI Bridge
DCI BRIDGE
WFI
DCI Bridge sends back job status to WFI.
WFI requests configuration
data for job submission from
WFS. From response WFI
creates a JSDL
1
3
JSDL
PLUGIN
DCI Bridge gets
inputs from
Storage
5
WFS
6
DCI Bridge
sends outputs
to the Storage
PLUGIN submits the job
together with inputs to
a DCI, monitors the job
status and gets the
outputs.
Storage
DCI
4
Steps of Plugin Creation - 1
1. Developing middleware specific parts
Implementing of 4 methods:
• submit (invoked on job submission; performs job submission
• getStatus (invoked when job status need to be queried; queries
job status and sets job status accordingly
• abort (invoked when the job need to be aborted; aborts the
execution of the job)
• getOutputs(invoked when the outputs of the job need to be
downloaded; downloads the outputs to a local folder)
Steps of Plugin Creation – 2
2.Developing configuration interface on DCI Bridge
New
Middleware
2.1 Adding new middleware name in mb_scheduling_description_language.xsd (other fields relevant for resource (VO,
DCI) selection are generic)
…..
<xsd:simpleType name="DCINameEnumeration">
<xsd:restriction base="xsd:string">
<xsd:enumeration value="glite"/>
<xsd:enumeration value="arc"/>
<xsd:enumeration value="unicore"/>
<xsd:enumeration value="boinc"/>
<xsd:enumeration value="gemlca"/>
<xsd:enumeration value="pbs"/>
<xsd:enumeration value="lsf"/>
<xsd:enumeration value="gae"/>
<xsd:enumeration value="service"/>
<xsd:enumeration value="local"/>
<xsd:enumeration value=”newmid"/>
</xsd:restriction>
</xsd:simpleType>
Steps of Plugin Creation - 3
2.2 Extending configuration schema with middleware-specific configuration
possibilities in dci-bridge_configuration_schema_2012.xsd
<xsd:element name="unicore" minOccurs="0" maxOccurs="1">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="keystore">
<xsd:simpleType>
<xsd:restriction base="xsd:string"/>
</xsd:simpleType>
</xsd:element>
<xsd:element name="keypass">
<xsd:simpleType>
<xsd:restriction base="xsd:string"/>
</xsd:simpleType>
</xsd:element>
<xsd:element name="keyalias">
<xsd:simpleType>
<xsd:restriction base="xsd:string"/>
</xsd:simpleType>
</xsd:element>
<xsd:element name="subjectdn">
<xsd:simpleType>
<xsd:restriction base="xsd:string"/>
</xsd:simpleType>
</xsd:element>
<xsd:element name="truststore">
<xsd:simpleType>
<xsd:restriction base="xsd:string"/>
</xsd:simpleType>
</xsd:element>
<xsd:element name="trustpass">
<xsd:simpleType>
<xsd:restriction base="xsd:string"/>
</xsd:simpleType>
</xsd:element> </xsd:sequence> </xsd:complexType>
</xsd:element>
2.3 Extending existing classes that process
tab/menu selection logic
2.4 Creating JSPs
Steps of Plugin Creation - 4
Creating new class
(JobConfigUI_newm
id)
Implementing the
getJsp and the
getJobParameters
methods.
WS-PGRADE
3.Developing WS-PGRADE and WFI specific parts
New middleware
Modification in WFI: Modify the JobConfig class: Adding the new
middlewares name in mbsdlMiddleware() method (it maps the job's
configuration to the middlewares configured in DCI-Bridge).
DATA BRIDGE
Outline
• Problem statement
• Data Bridge as independent DCI service:
– Data Bridge concept
– Use-cases
– Data Bridge architecture
• WS-PGRADE integration
– Data browsing portlet
• gUSE integration
Problem statement
• Scientific applications:
– Individual jobs or workflows
– Access data from diverse sources
– Science Gateways can hide the details, but…
• Data sources:
– Diverse types: HTTP, FTP, GridFTP, SRM, iRODS, …
– Thus, different APIs are needed to access these
• One possible solution is to use a service that can be
used to access the sources through a unified
interface
Data Bridge
• Offers a simple service that provides a generic
interface above different DCI's storage services
to handle the data stored
• The service in different use cases offers a way to
browse, upload and download data, and with the
help of multiple server instances it enables interDCI data transfer as well
Use cases
• Use case 1: Browse a single DCI data storage
from WS-PGRADE, upload data
• Use case 2: Transfer data files between
different DCIs
• Use case 3: Fetch input data on a DCI worker
node from an other DCI
• Use case 4: Cloud storage usage
Use case 1: Storage browsing and
data upload
WS-PGRADE
Storage Browsing
Portlet
Data Bridge
Adaptor Interface
Storage Adaptor
Storage
Browse
and
upload
Use case 2: Data Transfer – Using multilevel Data Bridge
Client:
•Storage Browsing Portlet
•Custom application
•…
Data Bridge
Adaptor Interface
Storage Adaptor1
Data Bridge Adaptor
Data Bridge
Adaptor Interface
Storage Adaptor2
Storage1
Storage2
Use case 3: Fetch data on a DCI’s worker node
from a „foreign” DCI’s storage
• Data bridge usage guidelines:
– First try to fetch the data using native tools
– Only if this fails, use the Data Bridge
DCI Worker node
Data Bridge
Wrapper
Adaptor Interface
Pre-process
Storage Adaptor
Executable
Storage
Post-process
Use case 4: Cloud Storage access
from WS-PGRADE/gUSE
• Currently, no S3 support in WS-PGRADE
• An S3 Data Bridge adaptor would fix this
DCI
WS-PGRADE/gUSE
Worker node
Job
Amazon S3
Data Bridge
Data Bridge Architecture
Public Interface
HTTP servlet
Adaptor Manager
Temporary URL queue
Worker Pool
Thread1
Thread2
Threadn
URI
URI
Adaptor Interface
DCI Adaptor1
jSAGA
DCI Adaptor2
DCI Adaptor3
DCI Adaptorm
URI
Data Bridge components
• Interfaces:
– Public Interface
– Adaptor Interface
• Adaptor Manager
• Worker Threads
• DCI Adaptors
Data Bridge componentsInterfaces
• Public Interface:
– Provides the public interface for external
components (Portlets, gUSE, …)
– Web Service interface
• Adaptor Interface:
– A Java interface that hides the details of the
different adaptors
Data Bridge Public Interface
• Operations:
–
–
–
–
–
–
–
List
Mkdir
Delete
Get
Put
Copy
Move
• Entities:
– URI (either a path, an URL or some specific class)
• Error reports:
– Common exceptions
Data Bridge Public Interface URI
• Represents an element with a given URI (a directory,
a file, metadata attributes, …)
• Also needs to carry security credentials (if needed)
• Attributes:
– Nothing special in the base class
– For gLite, e.g:
•
•
•
•
Path: the full path
Type: directory or file
Size: length of the entity (0 for directories)
Attributes: optional, contains information as returned by the
Adaptor Interface's Stat function
Data Bridge Public Interface –
Get and Put
• Two-phase up- and
download
withHTTP
theservlet
Public
Interface
temporary URL queue:
Adaptor
Manager
• First, the web
service
interface is invoked to
Temporary URL queue
register the transfer
request
Worker Pool
URI
URI
URI
• Next, a simple
HTTP client
Thread
Thread
Thread may use HTTP GET or
POST/PUT to down- or upload the data
1
2
n
• This way, web serviceAdaptor
invocation
Interface
(„heavyweight” SOAP) is separated from data
transfer („lightweight” HTTP)
DCI Adaptor1
DCI Adaptor2
DCI Adaptor3
DCI Adaptorm
Adaptor Manager and Worker
threads
• Provided by JAX-WS web service API
• Tasks:
– Manage incoming requests
– Initialize worker threads to perform the requested
operation
– With the help of different adaptors
DCI Adaptors
• Implement: Adaptor Interface
• Tasks:
– Perform operations requested by the Worker Threads, that
is operations invoked through the web service
• Types:
–
–
–
–
–
gLite (using jSAGA)
GridFTP (using jSAGA)
FTP (using jSAGA)
…
Data Bridge: special adaptor to forward requests to other
Data Bridges
Data Bridge clients
• Web Service clients:
– Create your own based on the WSDL (or REST)
• Java API:
– Provides a convenient tool to use Data Bridge
Public Interface functions
– Data transfer functions should accept InputStream
and OutputStream objects as their arguments
WS-PGRADE integration
• A Data Browsing portlet that eases storage
management
WS-PGRADE Workflow I/O
configuration
• During a workflow node's
IO configuration the user
should be able to select
files from storages
• The provided interface
should be the same as
the selected storage's
Storage Browsing portlet
(only with one panel)
Current status, future work
• Core Data Bridge (available as a web service)
ready, working with most major protocols
(FTP, GridFTP, SRM)
• User Interface development has been started,
first version will be available as part of WSPGRADE/gUSE shortly
ROBOT CERTIFICATE
The concept of robot
certificates
•
•
•
•
The normal certificate is used to identify users
The robot certificate is used to identify applications
As a consequence the application should be trusted
When the CA provides the certificate for the
application, the certificate contains the identifier of
the person or organization that validated the
application and takes the responsibility for it
• It is the policy of the user community and the CA to
decide whose name should be in the certificate
42
EGI VO Portal Policy I.
●
●
●
The Portal, the VO to which the Portal is associated,
the Portal manager are all individually and
collectively responsible and accountable for all
interactions with the Grid, unless a credential of a
Strongly Identified Web User is used to interact
with the Grid
The Portal must be capable of limiting the job
submission rate
The Portal must keep audit logs for all interactions
with the Grid
(https://documents.egi.eu/document/81)
EGI VO Portal Policy II.
●
Portal classes (in fact, these are working mode classes, i.e.
the same portal can be in parameter mode from the point
of view of a certain user and in the same time can be in job
management mode from the point of view of another user):
SZTAKI gUSE
EGI VO Portal Policy III.
●
●
Robot certificates can be used only for the first 3
working modes of the portal
Job management mode portals/applications must
not use robot certificates
SCI-BUS portals and EGI
• According to the EGI classification :
– WS-PGRADE/gUSE is a portal that can be used in 1,2,3,4
modes
– The community portals could also work in any of the 4
modes depending of the needs of the corresponding user
community
– Robot certificates are needed only for the 1,2,3 modes
46
Relationship between robot certificates and
WF applications in WS-PGRADE/gUSE
• The WF applications can have different robot certificates
• Even the jobs within a workflow can have different robot
certificates (this enables that different jobs of a WF can be
executed in different DCIs requiring different robot
certificates)
• This robot certificate contains the name of the community
who set up the gateway
• Example:
– Autodock gateway set up by SZTAKI and UoW
– The robot certificate will contain the name: SCI-BUS
47
WS-PGRADE/gUSE extensions to
support robot certificates
• The robot certificates should be hidden for the end-users but
manageable for the portal developer/administrator
• The WF applications with robot certificates will be stored in
the internal repository of gUSE
• Consequences:
– The internal repository should be extended to be able to store the
identification of the robot certificates for every node
– The portal developer/administrator (with a new privileged role to be
introduced besides the power and end-user roles) should be able to
assign the robot certificates for the WF nodes
48
Suggested process to assign
robot certificates to WF nodes
• When a WF is tested and ready to use in the community portal the next
step is to assign robot certificates to the nodes of the WF
• This will happen in the following way:
1. Portal developer imports the WF from the internal repository
2. Use the WF configuration facility of WS-PGRADE (this will be
extended to enable the definition of robot certificates, see next
slide)
3. Test the workflow with the assigned robot certificates
4. Export the WF with assigned robot certificates into the internal
repository
• Notice that some nodes of the WF can work with robot certificates while
other nodes require user certificate. Therefore even during the execution
of a WF the portal can change among working modes.
49
UI extension to assign robot
certificates to WF nodes
+ line (robot cert opt)
+ checkbox
+ auto ModWin
50
Usable Certificate types
X509 and Myproxy:
A:
B: DN and HOST cert
CloudBroker Platform:
A:
Unicore:
B: User’s own user/pass
A:
B: User’s own assertion
*
PBS or LSF cluster:
A: Generating new key
B: User’s own key
Storing the robot certificate
for a given job
• When the portal developer saves the WF that contains Job A
with robot certificate the following happens:
– The portal stores Job A with the required robot certificate
at a secure position. This stored job bundle will get a job
bundle identifier
– In the configuration of Job A this identifier will be add the
original configuration information
– When the workflow is saved to the repository this new
extended configuration field (containing the Job A bundle
identifier) will be saved
52
Job Bundle structure
• Binary (executable)
• Robot certificate parameters (depending on
the middleware type). This could be:
– Login/password (e.g. PBS, cloud)
– Certificate access information (e.g. gLite, GTx)
53
Execution of a WF with robot
certificates
• When the WFI interprets Job A with the robot certificate it will
place into the JSDL the Job A Bundle identifier, too
• When the DCI Bridge receives a job with a bundle identifier, it:
– Goes to the Bundle service and asks all the information
related to this bundle
– If the security information is a certificate access, then goes
to the credential provider service (in WS-PGRADE) by
passing the certificate access information as input and gets
back a certificate proxy
54
Further requirements for
gUSE to provide
• Suitable log to identify the person who
submitted a WF with robot certificate
• Restriction of the number of jobs submitted to
a certain VO during a certain time period
55
Suitable log to identify the
person who submitted a WF
• Each logged event must contain:
– A timestamp
– The portal user’s ID
– The portal user’s IP
• Job submit event should contain:
–
–
–
–
–
–
–
–
WF’s name
Job’s name
Job’s PID (in case of parametric jobs)
Job’s DCI Bridge ID
Credential used (proxy DN, username, …)
Input file list with sizes
Grid ID in case of successful submission
Error message in case of job submission failure
• These are already stored in DCI Bridge (see next slide)
Suitable log to identify the
person who submitted a WF
• Job status change events should contain:
– The job’s DCI Bridge ID
– The job’s new status
– Optionally: the job’s old status
• Terminal job status events should contain:
– Outcome (success, failure, with exit code)
– List and size of output files transferred
Log information in DCI Bridge
• After completing a job a zip file is created
identified by the job identifier and containing
all the information of the previous slide. This
contains 3 files:
– DCI Bridge log
– JSDL
– Log created in the DCI resource
• We will store these zip files in a temporary
storage area but archiving them will be the
responsibility of the portal admin
58
Restriction of the number of jobs
submitted to a certain VO
• This information will be stored in the DCI
Bridge
• This is a parameter configurable by the portal
admin for each DCI VO
• This information should be given when the
robot certificate is assigned to a certain a job
59