Developing with OMII
The Application Integration Tutorial
Stephen Crouch
Steven Newhouse
1
What we will cover…
The application deployment tutorial documentation
The example application we wish to install
The 5 steps to deploying an application



1.
2.
3.
4.

Create a subdirectory structure on the server
Write the application wrapper scripts
Add new application to the Resource Allocation Service
Add new application directory to the Job Service
Testing the deployment
2
The application deployment
tutorial documentation

Application deployment documentation
teaches how to deploy the mbio1 example
Java application

Can easily be generalised to installing any
application
3
The application deployment
tutorial documentation

Developer pre-requisites…




Basic shell script (Bash) expertise
Knowledge of overall OMII architecture
Familiarity with OMII Client command line
In context…
Application
Integration
Tutorial
Client
Command
Line Tutorial
Java Client
Tutorial
Services
Development
Tutorial
Being ported to
OMII 2…
4
Tutorial pre-requisites

Installation of an OMII Client (Windows/Linux)



Development platform
Used to submit jobs
Access to an OMII Server


With the OMII Basic Services Stack
An open, credit-checked account on the server
5
Start OMII Server
prompt> cd /home/omiiserv/OMII
prompt> ./database_start.sh start \
/home/omiiserv/OMII/postgres
prompt> cd jakarta-tomcat-5.0.25/bin
prompt> ./start_base.sh
6
The example application we
wish to install

The mbio1 application

Can be obtained from http://www.ecs.soton.ac.uk/
~stc/omii/training.html – Application Development Materials
 As root, unzip mbio1.zip in home directory:






Server subdirectory – materials for installing the application
Client subdirectory – materials for testing the install on client
Written in Java – mbio_1.java
Accepts single bio_1_data.txt file as input (fixed)
Outputs single mbio_1_results.csv as output (fixed)
Ensure Java environment set…
 Go to directory: cd mbio1/server
 Compile it: javac –classpath . mbio_1.java
 Run it: java –cp . mbio_1
7
Interface between the server
and executable
Job Service architecture
Used to
start mbio1
mbio1
Used to
check status
of mbio1
Not strictly
required
8
Step 1: create a subdirectory
structure


We refer to location of OMII server as
OMII_HOME
Applications have own directory stored in
OMII_HOME/demo


For mbio1, create a OMII_HOME/demo/mbio1
directory (e.g. mkdir OMII_HOME/demo/mbio1)
Then, create two subdirectories:
 mbio1/application – copy mbio1 class in here


e.g. cp mbio_1.class OMII_HOME/demo/mbio1/application
mbio1/wrapper – location of wrapper scripts
9
Step 2: Write application
wrapper scripts



Interface between server and application
Helps to understand the job run process
When a job run is requested from a client…

Service provider responsible for creating a workspace
directory for the running job
 Workspace in OMII_HOME/datastore/<date-time>_storeid
 Contains logs/status files for job
 Creates 2 subdirectories:



/stagedzips: input/output files moved to/from the staging areas
/work: temporary storage (scratch) area for running job
Service provider responsible for invoking start script
through the platform start script
10
Step 2: Write application
wrapper scripts
Application start script responsible for 3 things:

1.
2.
3.
Moving all input files from /stagedzips directory to /work
directory (unzipping, for example, if necessary)
Executing the application command
Preparing the output files for the client (e.g. zip the files),
and moving them to /stagedzips directory
Input/output file locations passed to start wrapper
script in following manner:


-i ../stagedzips/input0.zip -i ../stagedzips/input1.zip … -i
../stagedzips/inputN.zip -o ../stagedzips/output0.zip -o
../stagedzips/output1.zip … -o ../stagedzips/outputN.zip
11
Step 2: Write application
wrapper scripts

When job status is requested from a client…



Service provider responsible for invoking the application’s
status script through the platform status script
Application status script responsible for returning textual
status information to the client
 Perhaps to be parsed, or just output to the screen
Status file argument passed to status script:
 e.g.: -f .app_sp_status
 Any output from application status script is placed in this file
 File contents sent to client
12
Step 2: Write application
wrapper scripts – start script
#!/bin/sh
exec > log
echo mbio1 wrapper started
…
INPUT ="$2"
OUTPUT ="$4"
Pipe output to log file –
used for status script
Input/output file locations (in
stagedzips directory) passed from
platform script
echo Copying input to work directory ...
cp "$INPUT" bio_1_data.txt || exit 1
echo Run mbio1 ...
/usr/java/j2sdk1.4.2_04/bin/java –cp
/home/panel/OMII/demo/mbio1/application
 mbio_1 || exit 1
echo Copying result to output staging ...
cp mbio_1_results.csv "$OUTPUT" || exit 1
echo mbio1 completed successfully
1. Copy our staged input
to working directory
2. Location of
mbio_1 class –
WILL need to edit
it to this!
3. Copy our output to
stagedzips directory
13
Step 2: Write application
wrapper scripts – start script

Set execute permissions on the script:



Type: chmod ugo+x startJob
Copy startJob to OMII_HOME/demo/mbio1/wrapper
Can now test the script (a sensible check!):





cd to OMII_HOME/demo/mbio1/wrapper
Create subdirectory: mkdir stagedzips
Copy bio_1_data.txt to stagedzips
Run script:
 ./startJob –i ./stagedzips/bio_1_data.txt –o
./stagedzips/mbio_1_results.csv
mbio1_results.csv should be in /stagedzips directory
14
Step 2: Write application
wrapper scripts – start script

Look at log file: cat log

Should have contents like the following:
mbio1 wrapper started
Copying input to work directory
...
Run mbio1 ...
10
10
2.0
Copying result to output staging
...
mbio1 completed successfully
15
Step 2: Write application
wrapper scripts – status script
#!/bin/sh
cat log > $2
Output status file name path to use,
passed from platform status script
16
Step 2: Write application
wrapper scripts – status script

Set execute permissions on the script:



Type: chmod ugo+x checkJob
Copy checkJob to
OMII_HOME/demo/mbio1/wrapper
Can now test the script:


cd to OMII_HOME/demo/mbio1/wrapper
Run script:


./checkJob –f status_file.txt
Contents of log file should now reside in
status_file.txt
17
Step 3: configure the Job
Service

Individual applications form part of an
application suite and require unique names


There is a recommended convention…
Need to decide on unique application suite name


We’ll use http://omii.ac.uk/mbio1_suite
Our application suite only contains one
application – also requires a unique name

We’ll use http://omii.ac.uk/mbio1_suite/mbio1
18
Step 3: Add application to the
Resource Allocation Service

Need to inform Resource Allocation Service
of…



The existence of a new application suite
The machines that have it installed
We use the OMII Services web tools to do
this


Start a browser, and point it to:
http://<yourhost>:<yourport>/axis
Enter omii as the user, and admin as the
password
19
Step 3: Add application to the
Resource Allocation Service


Select ‘Resource Allocation Service’
First, need to add a new application suite:




Scroll down to “Application Suites” -> “Add a New
Application Suite”
Enter application suite URI
(http://omii.ac.uk/mbio1_suite) in the “Application
Suite URI:” field
Select “Add Application Suite”
Presented with application suite config page
20
Step 3: Add application to the
Resource Allocation Service

Next, need to add mbio1 application to
application suite



Scroll down to “Applications in
http://omii.ac.uk/mbio1_suite”
Enter application name
(http://omii.ac.uk/mbio1_suite/mbio1) in the
“Application Name” field
Select “Add Application to Suite”
21
Step 3: Add application to the
Resource Allocation Service

Need to specify machine name that has the
application suite installed



Scroll down to "Machines available for
http://omii.ac.uk/mbio1_suite”
Select the machine name, (name of the machine
on which the application is deployed), from the
"Select a Machine" dropdown list
Select "Add Machine to Application Suite"
22
Step 4: Add application to the
Job Service

Need to inform job service where application
wrappers are installed



Go back to http://omiitest12.omii.ac.uk:18080/axis
Select ‘Job Service’
Enter the directory for the newly added wrappers



E.g. /home/omiiserv/OMII/demo/mbio1/wrapper
Select "Add“
mbio1 application has been deployed!
23
Testing the deployment


As the omii user…
Copy the mbio1/client directory contents to
the OMII_CLIENT_HOME directory (e.g.
~/OMIICLIENT)


e.g. cp –r client/* ~/OMIICLIENT
Copy the Accounts.xml file generated earlier
to the mbio1 subdirectory as Account.xml:

e.g. cp test-services/test/Account-test.xml
mbio1/Account.xml
24
Testing the deployment

OMII_CLIENT_HOME should now contain:


The mbio1 subdirectory contains files used by script:



test-mbio1.sh: script to test deployment
bio_1_data.txt: input file to mbio1
Work.xml: example of individual job requirements to run a
single mbio1 job
Ensure java environment is set


export JAVA_HOME=/usr/java/j2sdk1.4.2_04
export PATH=$JAVA_HOME/bin:$PATH
25
Testing the deployment –
client test script test-mbio1.sh
DIR=./mbio1
java -cp src/java
uk.ac.soton.ecs.omii.client.utilities.CreateRequir
ementsFile http://omii.ac.uk/mbio1_suite
$DIR/Requirements.xml
./omii_client.sh tender $DIR/Account.xml
$DIR/Requirements.xml MyTask
./omii_client.sh upload MyTask $DIR/bio_1_data.txt
bio_1_data.txt
./omii_client.sh run MyTask
http://omii.ac.uk/mbio1_suite/mbio1 $DIR/Work.xml
--input bio_1_data.txt --output mbio_1_results.csv
./omii_client.sh download mbio_1_results.csv
mbio_1_results.csv
26
./omii_client.sh finish
Testing the deployment

Run the test in OMII_CLIENT_HOME directory as
user omii:
…
 ./test-mbio1.sh
Contacting
http://omiitest12.omii.ac.uk:18080/axis/services/JobService#58
http://omii.org/mbio1_suite/mbio1:
URL
:
http://omiitest12.omii.ac.uk:18080/axis/services/JobService#58
Status: Status is now output-staging-complete (FINISHED)
> JOB_STATUS
FINISHED
> mbio1 wrapper started
> Copying input to work directory ...
> Run mbio1 ...
> 10
> 10
> 2.0
> Copying result to output staging ...
> mbio1 completed successfully
Success.
OGRE Client
Downloading:
…
27