server
Tutorial 3:Deploying an application on the OMII
This document is designed to provide a tutorial for deploying a user application on the OMII_1
Server. This tutorial is targeted at developers who are familiar with running basic jobs from
the OMII_1 Client Distribution using the command line on Linux.
Document Scope
This tutorial will focus on deploying a basic java application named mbio_1 (see Appendix A
for the source). We will not concern ourselves with the details of what the application does.
Rather, what is important is that the application takes in an input file named bio_1_data.txt
(see Appendix B) and produces an output file named mbio_1_results.csv. The steps
described in this tutorial can be generalised to deploy any user application written in
programming languages that are supported by the environment where the OMII Server is
installed.
Prerequisites
The prerequisites for the tutorial are:
1. The installation of an OMII_1 Server which has the OMII Base, Extension and Services
installed.
2. The installation of an OMII_1 Client (either on a Windows/ Linux platform). This will be
used as a platform to submit jobs in order to test the newly deployed application.
The Five Steps:
To deploy a new application, these are the five steps involved:
1.
2.
3.
4.
5.
Create a subdirectory structure.
Write application wrapper scripts.
Configure the Job Service.
Add the new application to the Resource Allocation Service.
Restart tomcat.
Step 1: Create a subdirectory structure
To deploy your application on an OMII server, the following directory structure is
recommended to store the application executable and wrappers. Assuming your OMII server
is installed on <omii-server-home> (the default installation directory for <omii-server-home>
is /usr/local/OMII), create a subdirectory structure under <omii-server-home>/demo. The
following is the directory structure created for our mbio1 application.
<omii-server-home>
> demo > mbio1
>
>
application
wrapper
Copy your application executable, e.g. mbio_1.class and save it in the /application
subdirectory, i.e. <omii-server-home> /demo/mbio1/application.
The subdirectory <omii-server-home> /demo/mbio1/wrapper will be used to store the wrapper
scripts that we are going to write in the next step.
Step 2: Write application wrapper scripts
There are two application wrapper scripts that are required when deploying a new application.
They are (a) the application-wrapper and, (b) the application-status-wrapper. The first script is
used to drive the application executable and the second is used to check the status of the
application (see Why do we need application wrapper scripts? for further explanation). The
two scripts are to be stored under <omii-server-home> /demo/mbio1/wrapper.
The application-wrapper
We'll start with the application-wrapper script. Take a quick look through at the wrapper
written for mbio1 below (we named the script mbio1-wrapper.sh). Note that it is written in
Bash. Really, you can choose to use any scripting language you like provided it is supported
by the platform where your OMII server is installed.
#!/bin/sh
exec > log
echo mbio1 wrapper started
if [ "$1" != "-i" ]; then
echo mbio_1 requires exactly one input; exit 1
fi
if [ "$3" != "-o" ]; then
echo mbio_1 requires exactly one output; exit 1
fi
INPUT ="$2"
OUTPUT ="$4"
echo Copying input to work directory ...
cp "$INPUT" bio_1_data.txt || exit 1
echo Run mbio1 ...
/usr/java/j2sdk1.4.2_04/bin/java -cp /home/user/OMIISERVER11feb/demo/mbio1/
application mbio_1 || exit 1
echo Copying result to output staging ...
cp mbio_1_results.csv "$OUTPUT" || exit 1
echo mbio1 completed successfully
There are three tasks which an application-wrapper is expected to do. Before we look at
them, perhaps its useful to know that when a job is run on an OMII server, a workspace
directory is created for the job. (You can take a look at a workspace directory on the server in
<omii-server-home>/datastore. The last directory in the list corresponds to the last job that
has been run on the server.) A workspace directory contains all the status files for a job
(which we shall ignore here) and two subdirectories called /stagedzips and /work. The
directory /stagedzips is a temporary storage for input /output files that are to be moved to/from
the (remote) data staging areas. The transfer of files between a /stagedzips directory and the
data staging area is the responsibility of the server and we will not concern ourselves with this
here. The /work directory is the actual working directory at which a job is run. It is also the
directory where our application-wrapper is executed by the server.
Bearing the directory structure in mind, let's have a look at the three tasks an applicationwrapper should do:
Move all the necessary input files from the /stagedzips directory to /work
(i)
directory, prepare the input files in the form that can be taken in by your
executable, e.g. unzip them if necessary.
Execute the application command.
(ii)
(iii)
Prepare the output in the form that the client is expected to get, i.e. zip
them up if necessary. Move the output from the /work directory to /stagedzips.
To see how these tasks are implemented, let's go through the script we have seen at the
beginning of this section. To start with, the output from the execution of the script is redirected
to a file called log and we'll see later how this file is used by the application-status-wrapper.
exec > log
One important thing to note is that at the service provider, this script is driven by the platform
start-job script (see The Need for Platform Scripts) which feeds the application-wrapper with
input arguments that look like, for example -i ../stagedzips/input0.zip -o ../stagedzips/output0.
zip. The input and output file names are fed through to the platform script from the run job
command initiated by the client API.
In our case, it is the second and fourth input arguments which are required for our applicationwrapper, and they are saved as two local variables:
INPUT ="$2"
OUTPUT ="$4"
In the event that your application has more than one input or output file, the input arguments
that are passed to the application wrapper script will look like this:
-i ../stagedzips/input0.zip -i ../stagedzips/input1.zip ... -i ../stagedzips/inputN.
zip -o ../stagedzips/output0.zip -o ../stagedzips/output1.zip ... -o ../stagedzips/
outputN.zip
Note that when a file is uploaded to the OMII_1 server, it is always saved as a zip file with the
generic name inputN.zip or outputN.zip.
To run a job on the application, the script first copies the input file from the /stagedzips
directory to the /work directory. The /work directory is also the directory where the application
wrapper is run. So we have,
cp "$INPUT" bio_1_data.txt || exit 1
The || exit 1 is a robust feature where if the command fails, the script will terminate.
With the input file in place, we execute mbio1 by specifying the executable and the class path
pointing to the location of where mbio_1.class is installed on the server.
/usr/java/j2sdk1.4.2_04/bin/java -cp /home/user/OMIISERVER11feb/demo/mbio1/
application mbio_1 || exit 1
If the execution is successful, we copy the result to the /stagedzips directory, e.g.
cp mbio_1_results.csv "$OUTPUT" || exit 1
$OUTPUT denotes the output file and the path which it is to be copied to. Make sure the
script is executable by doing the following:
$ chmod ugo+x <omii-server-home>/demo/mbio1/wrapper/mbio1-wrapper.sh
Now, we can test the wrapper script. For the test to work, we may create a pseudo /
stagedzips directory under the directory where this wrapper script is saved, in our case its
<omii-server-home> /demo/mbio1/wrapper. Copy and save bio_1_data.txt in /stagedzips. We
are now ready to test the wrapper script, using this command:
$ ./mbio1-wrapper.sh -i ./stagedzips/bio_1_data.txt -o ./stagedzips/mbio_1_results.
csv
The above test is successful if you get mbio_1_results.csv in the /stagedzips directory. Check
that a file named log is generated and have a look at the content of the file using
$ cat log
In the case if an application take in more than one input or output file, the application-wrapper
may look like this:
#!/bin/sh
exec > log
...
INPUT0 ="$2"
INPUT1 ="$4"
INPUT2 ="$6"
OUTPUT0 ="$8"
OUTPUT1 ="$10"
OUTPUT2 ="$12"
echo Copying
cp "$INPUT0"
cp "$INPUT1"
cp "$INPUT2"
...
input to work directory ...
data1.txt || exit 1
data2.txt || exit 1
data3.txt || exit 1
echo Copying result to output
cp results1.txt "$OUTPUT0" ||
cp results2.txt "$OUTPUT1" ||
cp results3.txt "$OUTPUT2" ||
...
staging ...
exit 1
exit 1
exit 1
The application-status-wrapper
Now that we have written an application-wrapper, we can move on to write the applicationstatus-wrapper, which can be much simpler than the previous one. This script is called after
the job is submitted and the output from the script is fed back to the client for job status
update. Hence, you can write a script to display any useful message.
We now show you a very simple example of an application-status-wrapper for our mbio1
application. We named this file as mbio1-status-wrapper.sh and it is saved under <omiiserver-home> /demo/mbio1/wrapper.
#!/bin/sh
cat log > $2
As before, this script is driven by the platform getJobStatus script (see The Need for Platform
Scripts) which feeds the application-status-wrapper with two input arguments, i.e. -f ../.
app_sp_status. -f denotes the input file option. Any output message that we wish to display to
the client should be piped out to the .app_sp_status status file and the platform wrapper will in
turn read and display the status file to the client.
Again, make sure the application-status-wrapper script is executable by doing this:
$ chmod ugo+x <omii-server-home>/demo/mbio1/wrapper/mbio1-status-wrapper.sh
We can test our application-status-wrapper using the following command:
$ ./mbio1-status-wrapper.sh -f .app_sp_status
The contents of the log file generated by the application-wrapper previously will be piped to .
app_sp_status.
Step 3: Configure the Job Service
With the two wrapper scripts ready from the previous step, we can now edit the jobservice.
properties file located under <omii-server-home>/jakarta-tomcat-5.0.25/webapps/axis/WEBINF/classes to reflect where the scripts may be found.
First, backup jobservice.properties, and then edit the original file by adding the following two
lines at the bottom of the file:
jobservice.appscript.run.http\://omii.org/mbio1_suite/mbio1=<omii-serverhome>/demo/mbio1/wrapper/mbio1-wrapper.sh
jobservice.appscript.getStatus.http\://omii.org/mbio1_suite/mbio1=<omiiserver-home>/demo/mbio1/wrapper/mbio1-status-wrapper.sh
Here, we specify the location where the job service may find the two wrapper scripts for the
application. In this, we use a URI http://omii.org/mbio1_suite/mbio1 to refer to the mbio1
application. http://omii.org/mbio1_suite refers to the application suite that contains the
application mbio1. An application suite is a collection of applications which are used together
to achieve a specific task. In practice, any sensible URI may be used to name an application
suite and it does not reflect the location of the application suite. So if you were to type http://
omii.org/mbio1_suite on your browser, you will not be able to find anything. We adopt this
convention when naming a URI, i.e. http://<organisation>/<application-suite> with /
<application> as optional at the end of the URI.
In the event that you wish to add another application to an existing application suit, you will
need to create a new application-wrapper and an application-status-wrapper for the new
application and repeat Steps 1-5.
Step 4: Add the new application to the Resource Allocation Service
To add a new application to the OMII server, we use the resource allocation administration
(ra_admin) tool to add the application to an application suite, and inform the resource
allocation service that the application is deployed on the server.
To do so, first, start up a browser and go to http://<your-omii-server-hostname-and-domain>:
<port>/ra_admin. Log in as the resource allocation administrator using ra as the username
and admin as the password. There are a few steps involved in configure the ra_admin page.
First, we need to add a new application suite by doing the following:
Scroll down to "Application Suites" and go to "Add a New Application
Suite".
Enter your application suite URI, e.g. http://omii.org/mbio1_suite in the
"Application Suite URI:" field.
Select "Add Application Suite".
With the new application suite, we can now add your application to the suite by doing
these:
Scroll down to "Applications" and go to "Add a New Application to a Suite".
Select your application suite URI, e.g. http://omii.org/mbio1_suite from the
"Select an App Suite URI" dropdown list.
Enter your application name, e.g. http://omii.org/mbio1_suite/mbio1 (note
that the application name here is preceded with the application suite URI) in
the "Application Name" field.
Select "Add Application to Suite".
Lastly, we need to specify the machine which contains the application suite. This can be
done with the following steps:
Scroll down to "Application Suites Installed on Machines" and go to "Add
an Application Suite to a Machine".
Select the machine name, which is the name of the machine on which the
application is deployed from the "Select a Machine" dropdown list.
Select your application suite URI, e.g. http://omii.org/mbio1_suite from the
"Select an App Suite URI" dropdown list.
Select "Add Application Suite to a Machine".
-
Step 5: Restart Tomcat
In order for the configuration to take place, we need to restart the OMII server container. To
do so, go to <omii-server-home>/jakarta-tomcat-5.0.25/bin and execute
$./shutdown_base.sh
follow by
$./start_base.sh
The mbio1 application is now deployed on the OMII server and is ready to be tested using an
OMII Client.
Testing
To test if your newly deployed application is working correctly, you can do so using an OMII
Client which is installed either on a Linux/Windows platform. For our running example mbio1,
we will test using the command line API from an OMII client installed on a Linux platform.
For convenience purposes, we create a subdirectory called mbio1 under the OMII client home
directory, e.g. /home/joe/OMIICLIENT (the home directory will be referred to as <omii-clienthome> from now on). Below is a list of files which are required when submitting a job to
mbio1. They are stored under <omii-client-home>/mbio1:
i.
The input files
- There could be one or more input files (depending on the application). In our
example, we have one input file, i.e. bio_1_data.txt which content is made up of
the following three lines:
100
10
0.77
ii.
Account.xml
This is the account XML file obtained when you open an account with your
service provider. The service provider is the OMII server on which your new
application is deployed.
iii.
Requirements.xml
This is an XML file which specifies the requirements of the resource you
need from your service provider in order to run all your jobs. It can be
generated using the java class CreateRequirementsFile which takes in two
parameters: (i) your application suite name, e.g. http://omii.org/mbio1_suite,
and (ii) the path and name of the XML file to be generated. You may want to
modify this file manually to suit the requirements of your application.
iv.
Work.xml
This is an XML file which specifies the resource requirements to run a
single job.
The requirements detailed in this file will be checked against those in the
Requirements.xml file. If the requirements specified in the Work.xml file
exceeds the overall resource requirements detailed in Requirements.xml, your
job will not run. Hence, if you are running n jobs within the same allocation,
make sure the n times resources detailed in Work.xml does not exceed the
overall resource requirements detailed in Requirements.xml.
As the person in charge of deploying the application, you should make this
file available to users who wish to use your application.
Here is an example of a Work.xml file which details the requirement to run a
single job on mbio1:
<?xml version="1.0" encoding="UTF-8"?>
<spec xmlns:ns1="http://www.gria.org/org/gria/serviceprovider/
commontypes/messaging">
<ns1:work>
<ns1:std-CPU-seconds>50</ns1:std-CPU-seconds>
</ns1:work>
<ns1:min-physical-memory>
<ns1:bytes>200</ns1:bytes>
</ns1:min-physical-memory>
<ns1:max-output-volume>
<ns1:bytes>1000</ns1:bytes>
</ns1:max-output-volume>
<ns1:num-processors>1</ns1:num-processors>
</spec>
Running the test
Below is an annotated bash script which uses the OMII Client Command Line API (see OMII
Client Command Line Tutorial) to drive the whole process of running a job on the mbio1
application. The script is expected to be run from <omii-client-home>. In this particular
example, if the test is successful, you'll get an output file named mbio_1_results.csvin the
<omii-client-home> directory. Try to identify on your client screen the job status messages
that are produced by the mbio1-application-wrapper.
#!/bin/sh
## ===============================
## Submitting a job to mbio1.
## ==============================
# Assuming the current directory ./ is <omii-client-home>.
DIR=./mbio1
# Create Requirement.xml file that is required to request for resources.
java CreateRequirementsFile http://omii.org/mbio1_suite $DIR/Requirements.
xml
# Use a valid account specified in $DIR/Account.xml to request for
resources
# described in $DIR/Requirements.xml. This resource allocation will be
# referred to as MyTask.
./ogre_client.sh tender $DIR/Account.xml $DIR/Requirements.xml MyTask
# Upload the input file bio_1_data.txt to the service provider using the
resources allocated under MyTask.
./ogre_client.sh upload MyTask $DIR/bio_1_data.txt
# Run a job using the allocation MyTask on mbio1. Work.xml details the
# resources required by this job.
./ogre_client.sh run MyTask http://omii.org/mbio1_suite/mbio1 $DIR/Work.
xml --input bio_1_data.txt --output mbio_1_results.csv
# Download output.
./ogre_client.sh download mbio_1_results.csv
# Releasing resource allocation.
./ogre_client.sh finish
1.2 P1 © University of Southampton, Open Middleware Infrastructure Institute