LA1: Realizzazione del Polo di calcolo e sviluppo di nuove funzionalità di
GRID Computing.
Macoroobiettivo
Lo scopo di questa linea di attività è la realizzazione concreta di una griglia
computazionale definendo un insieme di strumenti standard capaci di realizzare in modo
uniforme l'accesso alle risorse informatiche disponibili, sia in termini di sistemi di calcolo
che di archiviazione di dati. Inoltre tali strumenti saranno in grado di garantire la sicurezza
di operazione della griglia e fornire la possibilità di monitorare in ogni istante il suo
funzionamento.
Le sedi ENEA maggiori sono dotate ognuna di un centro di calcolo che gestisce le risorse
installate localmente e che coprono esigenze di calcolo seriale, calcolo parallelo e
richieste di grafica avanzata. Il sistema ENEA GRID permette di accedere all'insieme di
tali risorse distribuite geograficamente, come ad un unico sistema virtuale, con una
capacità integrata globale di circa 3Teraflops, distribuita su sistemi paralleli (IBM SP,
cluster Linux, Cray XD1 e macchine SGI).
A partire da questa base operativa, tramite il presente progetto si svilupperanno
metodologie e soluzioni innovative, rispetto a quelle già in essere, al fine porre le basi e
sperimentare nuove soluzioni GRID che superino i problemi che attualmente sono irrisolti
o hanno una carente soluzione. Un particolare accento è rivolto alle integrazioni fra GRID
e gli altri progetti PON: PI2S2, CYBERSAR e ScoPE.
POSTER INDEX
1. Visualization of 3D laser scanner data from an archaeological site using OpenSceneGraph
in ENEA-GRID environment;
2. Cross checking OpenFOAM and Fluent results of CFD simulations in ENEA-GRID
environment ;
3. Image analysis of a nuclear plasma: Frascati Tokamak Upgrade using IDL and ENEAGRID technologies;
4. On the remote operation of a Transmission Electron Microscope;
5. New utilities and tools for using and monitoring ENEA-GRID environment;
6. Cresco computational resources and its integration in ENEA-GRID environment;
7. MAGO Monitoring All Grid Object;
8. New tools and technologies for the ENEA-GRID pervasive integration,
9. Advances in Interoperability between ENEA-GRID and gLitebased Infrastructures;
1
Visualization of 3D laser scanner data from an
archaeological site using OpenSceneGraph in
ENEA-GRID environment
D. Abate(1), M. C. Baracca(2), R. Ciavarella(2) , G. Furini(2), S. Migliori(2), S. Pierattini(2)
(1)
Universita’ di Chieti e Pescara, (2)ENEA
THREE-DIMENSIONAL SCANNING OF AN ARCHAEOLOGICAL SITE
One of the main issues to face in archaeology is the monitoring of a site during a period of
time, in order to document the preogress of the activity.This issue can be solved via
periodic 3D laser scanning of the site. In this way the whole area interested by
archaeological digs can be kept under control.
Each file obtained via 3D scanning contains millions of x,y,z coordinates, each
representing one point of the scanned archaeological site.A colour label in the RGB format
is associated to each point.
Usually, the final result of the process is a set of points, which may be displayed using the
OpenSceneGraph ibraries and ENEA GRID environment.
The 3D model obtained by laser scanning is an objective database which allows to extract
information about the site morphology, geometric structure and materilas.
If the database contains a sufficiently high number of points, significant degradation
phenomena can even be evaluated.Therefore, during post-processing activity on the 3D
database, it is possible to retrive plant, section and 2D elevation views.
Enea Laser Scanner rilief
at Juvanum Site (Montenerodomo, Chieti)
ENEA has a Time Of Flight (TOF) 3D Laser
Scanner (Leica Geosystem, HDS 3000).
HDS 3000 is a high speed and high precision laser
scanner combining different features suitable for a
wide range of applications.
Juvanum (Montenerodomo, Chieti)
The maximum instrument range is about 300
meters with an accuracy of 3 mm at 50
meters.The distance between each point and the
laser source is calculated according to the time
between the emission of the green transmitted
pluse and the return signal, that is recorded by a
detector.
The instrument was used by Enea technicians, INFO-GER unit of Bologna, to scan the
archaeological Juvanum roman site (Montenerodomo, Chieti), in cooperation with the
Department of Science of the “G. D'Annunzio” University (Chieti).
2
Set of points showing the RGB data
The
obtained
database
consists
of
16.589.664 points, each designated by x,y,z
coordinates, reflectance and colour data.
Subsequently, the database has been
optimized by removing the noise created by
trees, metal structures, people in the area
and various undesired objects.
After this post-processing phase, the final set
of points was reduced to 5.665.282 points.
VISUALIZATION OF A SET OF MILLIONS OF POINTS
The final database obtained as described above needed to be processed by an application
matching the archaeologist needs, i.e.:
 display of a large amount of data in a relatively short time (so allowing the operator
to see the cloud of points in a “near real time” manner while he is scanning the
archaeological site );
 operations on the points (e.g. distance between any two points, obtain plant views,
sections etc.).
Currently there are commercial software with the above features, but we were oriented to
an “open source” solution using our ENEA-GRID and the OpenSceneGraph libraries
(OSG), because this is the best for our purposes.
CONFIGURE OPENSCENEGRAPH IN THE ENEA GRID
3
The OpenSceneGraph main characteristics are:




Multiplatform (we have compiled OpenSceneGraph for Linux, Irix,Windows);
Supports OpenGL and 3D stereo viewer;
High performance;
Powerful 3D viewer (osgviewer) that lets us to view in a fairly short time sets of
several millions of points.
Any ENEA GRID user has a high level graphic library with the ability to create his own
viewer with the desired characteristics. However the OSG does not allow to:
 Measure the distance between two points;
 Extract the plan views;
 Extract the section views.
We have faced first the distance issue: the OSG can not allow to pick a single point and
memorise its coordinates, but this limitation has been overcome by integrating into the
viewer code the additional ANN library.
In particular:
 we have built up a database containing the coordinates of all the points from the 3D
laser scanner;
 we have superimposed to the displayed image an invisible surface;
 when the user clicks on the desired point the matching coordinates on the surface
are kept;
 to face the situation when the user clicks on a display point not associated to any
database entry, the point from the original 3D database is assumed to be the point
closest to the clicked one;
 the coordinates of the selected point are displayed;
ANN is a library written
in
the
C++
programming language
to support both exact
and
approximate
nearest
neighbor
searching in space of
various dimensions.It
was implemented by
David M. Mount of the
University of Maryland
and Sunil Arya of the
Hong Kong University
of
Science
and
Technology.
Osg point cloud viewer
4
Cross checking OpenFOAM and Fluent
results of CFD simulations in ENEA-GRID
environment
Fiorenzo Ambrosino1, Salvatore Raia, Agostino Funel, Salvatore Podda, Silvio
Migliori
1)ENEA-FIM - Via Vecchio Macello – Loc. Granatello, 80055 Portici (Napoli), ITALY. Tel.
+39 081 7723574
e-mail: fiorenzo.ambrosino@portici.enea.it; www.cresco.enea.it;
The OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox can simulate
anything from complex fluid flows involving chemical reactions, turbulence and heat
transfer, to solid dynamics, electromagnetics and the pricing of financial options.
OpenFOAM is produced by OpenCFD Ltd, is freely available and open source, licensed
under the GNU General Public Licence.
The core technology of OpenFOAM is a flexible set of efficient C++ modules. These are
used to build a wealth of: solvers, to simulate specific problems in engineering mechanics;
utilities, to perform pre- and post-processing tasks ranging from simple data manipulations
to visualisation and mesh processing; libraries, to create toolboxes that are accessible to
the solvers/utilities, such as libraries of physical models.
OpenFOAM is supplied with numerous pre-configured solvers, utilities and libraries and so
can be used like any typical simulation package. However, it is open, not only in terms of
source code, but also in its structure and hierarchical design, so that its solvers, utilities
and libraries are fully extensible.
OpenFOAM uses finite volume numerics to solve systems of partial differential equations
ascribed on any 3D unstructured mesh of polyhedral cells. The fluid flow solvers are
developed within a robust, implicit, pressure-velocity, iterative solution framework,
although alternative techniques are applied to other continuum mechanics solvers. Domain
decomposition parallelism is fundamental to the design of OpenFOAM and integrated at a
low level so that solvers can generally be developed without the need for any ’parallelspecific’ coding. (www.opencfd.co.uk/openfoam).
Here is an example of cross checking results of a CFD simulation between OpenFOAM
and Fluent.
The test case is a modification of the “lid driven cavity” case of the official tutorial
distribution of OpenFOAM.
5
Courant number:Co 
t U
x
Reynolds number:Re  dU
Fig. 1: Problem specification of the cavity test.
As Figure 1 shows the fluid motion is made by the moving wall at the top of the cavity;
Initially the case run with a Reynolds number of 10, where the Reynolds number is defined
above and where d and U are the characteristic length and velocity respectively and  is
the cinematic viscosity supposed to be 0.01m2s-1
Several meshes refinement have been considered (also for the studies of performances in
terms of scalability); in this particular example the mesh is a simple quadrilateral mesh of
41 x 41 cells.
The unsteady solution is obtained with a timestep choice that makes it sure the Courant
number is less than 1 everywhere in the flow domain. The Courant number is defined for
one cell as is reported above where dt is the time step, U is the velocity through the cell
and dx is the cell size in the direction of the velocity.
We chose dt based on the worst case that correspond to dt = 0.005s.
Fig. 2: Velocity vector field of the cavity case.
Figure 2 shows the velocity vector map coloured by velocity magnitude calculated by
Fluent (vector length is constant and not related to velocity magnitude).
6
Fig. 3 and Fig. 4: Comparison of velocity profiles along the imaginary vertical middle
surface of the cavity (at x=0.05m) between Fluent and OpenFOAM
Figures 3 and 4 show a comparison of velocity profiles along the imaginary vertical middle
surface of the cavity (at x=0.05m) obtained from simulations based on OpenFOAM and
Fluent implementations of the cavity case. The latter was created manually with Fluent and
the former is provided by the OpenFOAM tutorial.
The Fluent implementation of the cavity case was converted in its OpenFOAM equivalent
one. To make an automatic conversion of the mesh we used the utility fluentMeshtoFoam
provided by OpenFOAM while manual changes were necessary to convert settings of
boundary conditions, initial conditions, physical characteristics of the fluid (viscosity etc.)
and the characteristic parameters of the numerical simulation.
The case obtained by this conversion was found to be fully equivalent to the original case
provided by the OpenFOAM tutorial.
We are now studying the capabilities and limitations for converting to OpenFOAM CFD
cases made by Fluent. The goal is to make this procedure as much as possible robust,
efficient and automatic on the ENEA-GRID.
OpenFOAM parallel performances
On numerical fluid dynamics, computational cost plays a crucial role. Many phenomena
can’t be treated with approximate numerical methods but by others more complex, such as
RANS (Reynolds Averaged Navier Stokes) for example, requiring the adoption of
resolution meshes need a large amount of computational calculation.
It is therefore important to have a calculating tool that can handle very hard CFD
simulations in reasonably execution time. In the last years computing power of processors
has reached a technological limit and then the technological solution adopted to solve
these CFD problems is to divide the work among parallel processors (parallel working).
7
For a CFD solver therefore it is important to be scalable or, in other words, to have a
significant improvement in performance as the number of processors used to solve the
problem.
Time [s]
Figure 5 shows the performances of
parallel simulations on a modification of the
Performances of OpenFOAM
case previously discussed (cavity: 2D,
65536
1000x1000 cells, laminar, incompressible,
32768
unsteady) obtained by varying the number
OpenFOAM - 64bit
16384
of cores.
8192
As ENEA users benchmarking group, we
4096
have used the EFDA-itm.eu cluster
2048
(http://www.efda-itm.eu) composed of 3
1024
front-end and 16 worker multi-core nodes.
512
Each node has 2 processors Dual-Core
256
AMD Opteron(tm) Processor 2218 (4
1
2
4
8
16
32
64
n: number of cores
cores) 2.6GHz, 16GB memory, Gigabit and
Infiniband
interconnections.
This
Fig. 5: Performances of OpenFOAM: architecture is very similar to the
scalabilty.
architecture of CRESCO.
Many simulations have been done by enabling or not the Infiniband interconnection (IB)
to show the improvements due to this powerful technology.
Figure 6 shows the speedup (obtained respect to the serial simulation) in both cases
obtained by using Gigabit and IB interconnection.
Simulations using IB have been done with an implementation of OpenMPI
(www.openmpi.com) that use the OpenFabrics (www.openfabrics.org) specification
software of Infiniband.
speedup
80.0
70.0
60.0
T(1)/T(n)
50.0
40.0
Gbit
30.0
IB
20.0
10.0
0.0
0
10
20
30
40
50
60
70
n: number of cores
Fig. 6: Performances of OpenFOAM: speedup variation due to interconnection.
8
Fig. 7: Performances of OpenFOAM: Gain in performances due to Infiniband interconnection rather than Gigabit.
Figure 7 shows the gain on the simulation time obtained by choosing IB rather than Gigabit
interconnection (we want to remember the reader that each node has 4 cores therefore the
number of nodes involved in the simulation is obtained by dividing the number of core by
4).
References:
www.fluent.com;
OpenFOAM User Guide; www.opencfd.co.uk/openfoam;
www.openmpi.com
9
Image analysis of a nuclear plasma:
Frascati Tokamak Upgrade using IDL and
ENEA-GRID technologies
M. Chinnicia, S. Migliorib, R. De Angelisc, S.
Borionid, S.
Pierattinie
INTRODUCTION
Today a number of applications in scientific fields (such as medical industry, astronomy,
physics, chemistry, forensics, remote sensing, manufacturing, defense) rely upon images
to store, display, and provide information about the world around us. The challenge to
scientists, engineers and business people is to quickly extract valuable information from
raw image data. The primary purpose of our work (within CRESCO Project) - i. e.
converting images of a nuclear fusion plasma coming from the experiments (shots) of
Frascati Tokamak Upgrade (FTU) into information (Fig.1) by ENEA-GRID
infrastructures– is related to such issue. In particular, we use IDL (Interactive Data
Language) in order to quickly access image data and to process them. IDL is a high-level
programming language that contains an extensive library of image processing and
analysis routines.
Fig.1 An internal view of the Frascati
Tokamak Upgrade vessel.
The limiter is shown in the central
part of the chamber.
SETTING
In modern tokamaks visible and infrared video cameras are becoming more and more
important to monitor plasma evolution during fusion experiments. The real-time analysis of
FTU images performed by IDL applications (Falsecolor, Database, Volume, Brightzone)
can really provide relevant information to control the plasma and the safety of the
machines.
Problem: In the last years video cameras have been extensively used in magnetic
confinement fusion experiments for both the understanding of the physics and the safety of
10
the operation. Both visible and InfraRed (IR) images can be used not only to monitor the
evolution of a plasma discharge but also to evaluate specific parameters, from the
determination of impurity radiation to the distribution of power loads on the plasma facing
components. Data analysis is normally performed off-line, due to the high amount of
information to be processed, making the data acquired by the camera quantitatively useful
only for post pulse evaluations. The main difficulty in using visible or infrared images for
plasma feedback control is the fact that real-time image processing is challenging and
heavy in terms of processing time, especially when complex tasks are required.
Novelty: At the beginning, the visualization of FTU images has been done under the
Videoftu Project. Since FTU image database is rather huge (4×106 Frames), we used the
multicase submission with multicluster queue to achieve efficient performance in terms of
elapsed time and CPU time. In order to reduce the run-time of the processes, the route of
multicase processing has been utilized.
IDL AND ENEA - GRID
In CRESCO Project, under the task “Development and Integration of the GRID and 3D
Graphics” we ported a number of applications which analyse and elaborate the images
coming from the tokamak database.
Goal: Image and processing analysis of FTU data through IDL applications: Falsecolor,
Database, Volume, Brightzone. In details, the applications allow image quality
improvement (noise reduction, contrast enhancement, distortions correction), automatic
classification by pattern recognition algorithms and brightness analysis, used to detect
images with a characteristic feature (quite recurrent in the plasma) in the brightness
distribution.
An example is the detection of bright toroidal bands (i.e. lying in the vessel’ s equatorial
plane), which precede the onset of regimes of enhanced gas recycling on the wall (a
phenomenon known in tokamaks as ‘Marfe’), sometimes followed by distructive events. A
second example is the identification of bright spots, characterised by typical shapes and
localization, which are due to high energy electrons (‘runaway electrons’), potentially
dangerous for the vacuum chamber. The applications allow a large number of tokamaks
images’s classification according to specific events and help understanding their
correlation with other physical quantities. On the other hand the achievement of event
recognition on timescales shorter than those of the evolution of unwanted events, can
provide a useful input for the feedback control of plasma operations
Methods: The experimental evaluation of the algorithm in IDL environment has been
performed through the use of the ENEA-GRID infrastructures for the submission and the
execution of jobs (Fig. 2-3).
We used the multicluster queue to achieve efficient performance in terms of elapsed time.
Hence, the experimental evaluation of the algorithm in IDL environment has been
performed through the use of the ENEA-GRID infrastructures for the submission and the
execution of jobs.
11
EXAMPLE OF APPLICATION DESCRIPTION
Fig.2 FTU images are input database (4×106 Frames). We have used the IDL resources in ENEAGRID infrastructures for the submission and the execution of jobs. In details, we used the multicase
submission with multicluster queue that run applications simultaneously on the 6 ENEA-GRID
clusters (Portici, Frascati, Bologna, Trisaia, Brindisi, Casaccia) in order to achieve efficient
performance in terms of ELAPSED and CPU time: benchmark tests we carried out on different
platforms with different type of queue show real and meaningful performance improvements in
running jobs by opting for this scheduling solution. The images analysis of FTU data through IDL
applications are in the output.
The utilization of the ENEA-GRID
tecnology is an efficient solution to reduced the run-time
required to execute the simulations.
12
IDL and ENEA-GRID submission: Experimental result with multiclaster queue
Fig.3 Benefits of multicluster queue. For example, let’s consider an FTU experiment where a rangeof
20 shots (each shot contains 109 frame) is produced by a single job:
CPU TIME : ≈ 38 min
Experimental Tests with distributed run:
ELAPSED TIME (10 parallel jobs): ≈ 91 hours
ELAPSED TIME (20 parallel jobs): ≈ 53 hours
REFERENCES
R. De Angelis, S. Migliori, S. Borioni, G. Bracco, S. Pierattini, A. Pierozziello, Analysis of images from
videocameras in the FTU tokamak, Review of scientific instruments, Vol. 75, N. 10
Idl reference guide, Vol. 1 and Vol. 2
http://ftu.frascati.enea.it
http://www.cresco.enea.it/LA1/cresco_ sp12_graf3d/
E-mail: marta.chinnici@portici.enea.it
a
ENEA-FIM, Portici Research Center, Via Vecchio Macello - Loc. Granatello - 80055 Portici (Naples)
ENEA-FIM, Enea-Sede, Lungotevere Thaon di Revel n. 76, 00196 Roma
c
ENEA, Frascati Research Center, Associazione EURATOM-ENEA sulla Fusione, Via Fermi ,00044,
Frascati
d
ENEA- FIM-INFOGER, Casaccia Research Center, Via Martiri di Monte Sole n. 4, 40129 Bologna
b
13
On the remote operation of a
Transmission Electron Microscope
M. Vittori Antisari, S. Migliori, L. Capodieci, R. De Angelis, G. Elmo, M. Nacucchi,
M. Palmisano, E. Pesce, E. Piscopiello, M. Re
ENEA
PROJECT AIMS
TECHNICAL GOALS
Allow full control of a transmission
electron microscope by a remote user
connected by internet in order to:
 set up collaboratory research;
 provide microscope time to trained
users in ENEA;
 and in research institutions;
 training new microscope operators by
e-learning;
 classroom teaching.
IN THIS WAY IT WILL BE POSSIBLE
 to share the TEM (Transmission
Electron Microscope) with other
groups
setting
up
scientific
cooperations;
 to optimize the capital investment and
share a fraction of the running costs.
FROM THE OPERATIVE POINT OF
VIEW
 integration
of
the
transmission
electron microscope as a node of the
ENEA-GRID;
 use of state of the art software tools
for the fast transfer of data and
images;
experimental testing to adapt the
configuration to the available bandwidth.
14
New utilities and tools for using and
monitoring ENEA-GRID environment
Authors: Agostino Funel, Guido Guarnieri, Salvatore Raia, Fiorenzo
Ambrosino, Silvio Migliori, Salvatore Podda
e-mail contact: agostino.funel@portici.enea.it,
guido.guarnieri@portici.enea.it, salvatore.raia@portici.enea.it
www.cresco.enea.it
INTRODUCTION
Under the CRESCO project, tools have been developed for supporting and improving ENEAGRID functions. Since one of the main goals is to achieve a system that users can treat as a
single virtual machine for High Performance Computing, we have tested and/or developed a
layer of tools that cope with heterogeneity, networks, hardware and software performances
and dynamic evolution of the grid state.
1. STATUS MONITORING AND NETWORKING
While data are sent and received by means of internet transport protocols like UDP and TCP,
a remote user or application can start its processes on a host of the ENEA-GRID by using
several authentication protocols like rsh, ssh etc. Therefore it is important to test both network
performance and authentication procedures. We present some of the developed tools: the rsh
test, the UDP test and the TCP/IP bandwidth test. The former is an example of a software
which is able to test the efficiency of a command over the full grid, the second measures the
rate at which data can be transmitted over the internet, the third measures the TCP/IP
bandwidth.
Fig. 1.1. From each host it is possibile to test the
status of the grid. The connectivity status of a given
host whit respect to the others can be tested by an
application which can be started, independently of the
host the user is logged on, by means of the ENEA
submission tool. This technique is important, for
example, to discover failures of the authentication
protocols (like rsh, ssh, etc...) used by many
applications based on parallel programming.
Fig. 1.2. The NRT (Network Response Time) tool has
been developed in order to detect how the
communication speed between two hosts of ENEAGRID changes over time. The network response time
is defined as the round trip time it takes to send a
packet of data and receive it back. The test uses the
UDP protocol. The average round trip time ‹ T› as well
as other statistical information are reported.
15
Fig. 1.3.The tool measures the network TCP/IP
bandwidth between two hosts (clent and server) of
ENEA-GRID. The user can specify the interval of the
band to analyse along with the step size. This tool
provides an efficient way to check whether the network
is working accordingly to the aspected value of the band
saturation point as well as to reveal data flow
bottlenecks.
2. SERIAL AND PARALLEL LAUNCHERS
ENEA-GRID is a collection of many clusters with distributed and shared memory
architectures. Hosts with operating systems like Linux, AIX and SGI IRIX, share a distributed
file system (AFS). Below are shown the different kinds of compilers (FORTRAN 77/90/95, C,
C++) and parallel programming resources based on Message Passing Interface model.
Fig. 2.1. Left: available MPI resources in ENEA-GRID. Right: Compilers present in ENEAGRID. Note that the same host may have installed more than one compiler.
Due to its heterogeneous environment, running an application with many binaries or
installations, one for each platform, requires job lauchers or wrappers. Their task is to check
the platform type from which a given application was launched, select the appropriate binary
and run it on the initial host. Fig. 2.3 shows the scheme we have adopted to hide the complex
structure of ENEA-GRID to users.
16
Fig. 2.3. Red lines trace the steps performed by a
command issued on a given platform in ENEA-GRID
In a grid environment jobs should be run under the control of a resource manager and thus
job launchers give to programmers the choice of submitting applications in interactive or
batch mode. Fig. 2.4 shows an example: regardless of serial or parallel jobs, many different
applications can be integrated in a single interface.
Fig. 2.4. Example of ENEA-GRID interface, Java classes
allow to associate a command or a chain of job launchers.
3. PARALLEL PROGRAMMING IN MATLAB
Matlab(MATrix LABoratory) provides an environment for numeric elaborations and an
interpreted programming language widely used in science and technology. The package is
equipped with specialized libraries (toolboxes) which can solve problems in several fields.
For some years the implementation of parallel programming in Matlab environment has been
under study. Mathworks, the producer of Matlab, distributes toolboxes with this target, but
there are also third party projects with the same aim. Among them are the “MatlabMPI” and
“pMatlab” libraries developed by Lincoln Laboratories of the Massachusetts Institute of
technology. ENEA-GRID with the integration of the CRESCO supercomputer, will achieve a
very high computational power. So, the possibility of parallel programming in Matlab
becomes a meaningful aim. We have chosen to modify and install the “MatlabMPI” library for
Linux machines of ENEA-GRID.
MatlabMPI. This functions library, written in pure Matlab language, implements a subset of
the functions defined in the MPI standard and allows to run, at the same time, several Matlab
processes both on multicore machines and computer clusters. Some code changes were
required to install MatlabMPI in ENEA-GRID. To enable the launching of Matlab jobs on
remote machines, allowing them to access the AFS file system without losing reading and
writing permissions, we have resorted to the ENEA-GRID submission tool. Data
exchange among processes does not happen by real message sending, but via access
operations to the common file system.
17
Fig.3.1 Plots show the results
of a test made with MatlabMPI
library for ENEA-GRID.
The involved machines are
lin4p, bw305-2 of Frascati and
prometeo0 of Casaccia clusters. The test consists of
measuring the elapsed time in each “send” and “receive”
sequence employed in the exchange of increasing
dimension messages among processes. This test does not
measure the pure bandwidth because send and receive
operations involve file system accesses. We can define it
as a test for the communication speed among processes
using MatlabMPI.
4. PERFORMANCE TESTS
HPL is a software package that solves a dense linear system in double precision (64 bits)
arithmetic on distributed-memory computers. It provides a testing and timing program to
quantify the accuracy of the obtained solution as well as the time it took to compute it. We
have used HPL to measure the performance of two kind of systems in ENEA-GRID: a Linux
cluster 16 nodes with 2 Daul-Core AMD Opteron per node, interconnected via Infiniband (IB)
and 1 SMP node Power 4 with 32 CPU. Fig. 4.1 and Fig 4.2 show the collected data. We
report the following main parameters:
1.
Memory. It is related to the dimension N of the matrix. Growing values of N allow a
better utilization of memory hierarchy, from local to level 1 chache memory. Moreover,
since HPL uses BLAS libraries, the algorithm can use Level 3 BLAS due to the twodimensional Block-Cyclic Distribution Data.
2.
Peak. It is the most significant benchmark result since it gives the number of floating
point operation per second the system can perform. We report it as a function of
memory utilization (or N).
Fig. 4.1. Performance Peak (Gflops) of the Linux
cluster. The Peak=48.12 Gflops produce a
efficiency of 83.50%.
18
Fig. 4.2. Performance Peak (Gflops) for Aix. The
Peak=90.06 Gflops produce a efficiency of about
50%.
References.
1 - UNIX Network Programming, Volume 1, Second Edition, Prentice Hall, 1998
UNIX Network Programming, Volume 2, Second Edition, Prentice Hall, 1999
2 - http://www.open-mpi.org/
http://www.afs.enea.it/project/eneagrid/Resources/List.html
3 - http://www.mathworks.com
http://www.ll.mit.edu/MatlabMPI
H. Kim, J. Mullen, Introduction to Parallel Programming and pMatlab, MIT Lincoln
Laboratory.
4 - http://www.top500.org
http://www.netlib.org/benchmark/hpl
19
CRESCO COMPUTATIONAL RESOURCES
AND ITS INTEGRATION IN ENEA-GRID
ENVIRONMENT
G. Bracco, S. Podda, S. Migliori, P. D'angelo, A. Quintiliani, D. Giammattei, M. De
Rosa, S. Pierattini, G. Furini, R. Guadagni, F. Simoni, A. Perrozziello, A. De Gaetano,
S. Pecoraro, A. Santoro, C. Sciò, A. Rocchi, A. Funel, S. Raia, G. Aprea, U. Ferrara, F.
Prota, D. Novi, G. Guarnieri.
ENEA, Lungo Tevere Thaon di Revel, Roma
Abstract
The paper describes the architecture of the high performance computing (HPC) system
that has been installed to provide the required computing power to the CRESCO project
applications and the dedicated activity required to integrate CRESCO HPC system into
the already existing ENEA-GRID infrastructure.
CRESCO HPC system consists of more then 2700 computing cores, divided into three
main sections. A section is dedicated to applications with high memory requirements ( 42
nodes with 16 cores and 32 or 64 GB memory for a total of 672 cores), a section
dedicated to high scalable applications (256 nodes with 8 cores and 16 GB memory, for a
total of 2048 cores) and a third experimental section providing systems with Cell
processors (4 blades), FPGA (6 VIRTEX systems) and high performance video adapters
(4 NVIDIA FX 4500 X2 systems) dedicated to computational applications. High bandwidth
and low latency connections are provided by an InfiniBand 4xDDR network. The main
storage consists of an IBM/DDN 9550 system with 160 TB raw data, organized in a GPFS
file system.
CRESCO HPC system has been integrated into ENEA-GRID infrastructure which has
been developed to provide a unified environment for all the main ENEA HPC resources.
The main software components of ENEA-GRID are the multi-site resource manager LSF
Multicluster, the OpenAFS distributed file system, the integrated Kerberos 5 authentication
and a Java and Web based Graphical User interface making use of CITRIX technologies.
The choice of mature, reliable and multi-platform software components has permitted
along the years to integrate in a GRID oriented infrastructure HPC resources at the state
of the art performances, with minimal changes in the user environment.
Introduction
ENEA, the Italian agency for the energy, environment and new technologies, has a
substantial experience in GRID technologies and its multi-platform HPC resources are
integrated in the ENEA-GRID infrastructure.
This paper describes the architecture of the high performance computing (HPC) system
that has been installed to provide the required computing power to the CRESCO project
applications and the dedicated activity required to integrate CRESCO HPC system into
ENEA-GRID infrastructure.
20
CRESCO (Computational Research Center for Complex Systems) is an ENEA Project, cofunded by the Italian Ministry of University and Research (MUR). The project is functionally
built around a HPC platform and 3 scientific thematic laboratories:



the Computing Science Laboratory, hosting activities on HW and SW design, GRID
technology and HPC platform management
the Computational Systems Biology Laboratory, with activities in the Life Science
domain, ranging from the “post-omic” sciences (genomics, interactomics,
metabolomics) to Systems Biology;
the Complex Networks Systems Laboratory, hosting activities on complex
technological infrastructures, for the analysis of Large National Critical
Infrastructures
CRESCO HPC system consists of more then 2700 computing cores, divided into three
main sections. A section is dedicated to applications with high memory requirements ( 42
nodes with 16 cores and 32 or 64 GB memory for a total of 672 cores), a section
dedicated to high scalable applications (256 nodes with 8 cores and 16 GB memory, for a
total of 2048 cores) and a third experimental section providing systems with Cell
processors (4 blades), FPGA (6 VIRTEX systems) and high performance video adapters
(4 NVIDIA FX 4500 X2 systems) dedicated to computational applications. High bandwidth
and low latency connections are provided by an InfiniBand 4xDDR network. The main
storage consists of an IBM/DDN 9550 system with 160 TB raw data, organized in a GPFS
file system.
CRESCO HPC system has been integrated into ENEA-GRID infrastructure which has
been developed to provide a unified environment for all the main ENEA HPC resources.
The main software components of ENEA-GRID are the multi-site resource manager LSF
Multicluster, the OpenAFS distributed file system, the integrated Kerberos 5 authentication
and a Java and Web based Graphical User interface making use of CITRIX technologies.
The choice of mature, reliable and multi-platform software components has permitted
along the years to integrate in a GRID oriented infrastructure HPC resources at the state
of the art performances, with minimal changes in the user environment.
21
CRESCO HPC system
CRESCO HPC system has been designed with the aim of offering a general purpose
facility based on the leading multi-core x86_64 technology.
The performance for the CRESCO HPC plant set-up has ranked #180 in the Nov. 2007
top500 list with Rmax=9.3 TeraFlops (rank #3 between the Italian HPC systems in the list).
In order to provide the best environment for different types of applications the system
consists of two main sections respectively oriented (1) for high memory request and
moderate parallel scalability and (2) for limited memory and high scalability cases. Both
sections are interconnected by a common Infiniband 4X DDR network (IB) and can
operate as a single large integrated system.
The first main section is composed by 42 fat nodes IBM x3850-M2 with 4 Xeon Quad-Core
Tigerton E7330 processors (2.4GHz/1066MHz/6MB L2), 32 MB RAM (4 extra-fat nodes
with 64 GB RAM). The total number of cores in the first section is then equal to 672.
The second main section is composed by 256 blades IBM HS21 each supporting dual
Xeon Quad-Core Clovertown E5345 processors (2.33GHz/1333MHz/8MB L2), 8 GB RAM
(16 blades with 16 GB RAM) for total of 2048 cores. The blades are hosted by the14 slots
blades chassis for a total of 19 chassis and each blade has a dedicated IB connection.
The larger system created by joining the two main sections is has 2720 cores.
A third experimental section consists of 3 subsections dedicated to special processor
architectures:
4 blades IBM QS21 with 2 Cell BE Processors 3.2 Ghz each.
6 nodes IBM x3755, 4 sockets AMD Dualcore 8222 equipped with a FPGA VIRTEX5
LX330 card
 4 node IBM x 3755, 4 sockets AMD Dualcore 8222 with a NVIDIA Quadro FX 4500 X2
video card


The IB network is based on a CISCO SFS 7024 (288 ports), a CISCO SFS 7012 (144
ports) and 5 CISCO SFS 7000 (120 ports) and its architecture is shown in fig. 1.
The Ethernet network consists of one CISCO 4506 (240 ports), 3 CISCO 4948 (144 ports)
and 3 CISCO 3750G (144 ports).
The storage of CRESCO HPC system is provided by an IBM DCS9550 system, 160 TB
raw space based on 500 GB SATA Hard Disk. An IBM Tape Library IBM TS3500 provides
the backup facility.
The power required to run the system has been estimated to 150 kw and proper cooling
systems have been provided.
The operating system is RedHat EL 5.1 and the usual set of Portland and Intel Compilers
are available.
A GPFS parallel file system is shared via Infiniband between all the computing nodes of all
the main section of the system. User homes are located in an OpenAFS file system, one of
the base elements of the ENEA-GRID infrastructure.
22
The three sections together with other 35 service machines (front-end, controls, fileservers, installation servers) and storage and network components make use of a total of
18 standard racks (19”, 42 U).
Fig.1
Architecture of the InfiniBand network including the IBM/DDN 9550 storage system.
The 4 I/O Nodes, directly FC attached to the storage, are the GPFS NSD servers
ENEA-GRID
ENEA, The Italian National Agency for Energy Environment and New Technologies, has
12 Research sites and a Central Computer and Network Service with 6 computer centres
managing multi-platform resources for serial & parallel computation and graphical post
processing.
Fig.2 : ENEA centers in Italy with 6 ENEA-GRID sites
23
ENEA GRID mission (started 1999) is focused to:


provide an unified user environment and an homogeneous access method for all
ENEA researchers and their collaborators, irrespective of their location.
optimize the utilization of the available resources
GRID functionalities of ENEA-GRID (unique authentication, authorization, resource access
and resource discovery) are provided using “mature”, multi-platform components:



Distributed File System: OpenAFS
Resource Manager: LSF Multicluster [www.platform.com]
Unified user interface: Java & Citrix Technologies
These components constitute the ENEA-GRID Middleware.
OpenAFS
 user homes, software and data distribution

integration with LSF

user authentication/authorization, Kerberos V
ENEA-GRID computational resources
Hardware (before CRESCO HPC system!!):
~100 hosts and ~650 cpu : IBM SP; SGI Altix & Onyx; Linux clusters 32/ia64/x86_64;
Apple cluster; Windows servers. Most relevant resources:
IBM SP5 258
cpu; 3 frames of IBM SP4 96 cpu
Software:
Commercial codes (fluent, ansys, abaqus..)
Research codes. (mcpn/x, eranos, fluka...
Elaboration environments (Matlab, IDL, Scilab...)
Windows Applications
ENEA GRID User Interface
ENEA GRID makes use of Citrix Metaframe to publish an application providing all the
available resources and monitoring facilities with a unified GUI interface Fig.3 .
GUI Application components:
Java (GUI)
shell script
24
Fig.3 ENEA GRID User Interface
ENEA-GRID Network connections
ENEA computational resources are distributed over WAN, connected by GARR, the Italian
Academic & Research Network (Fig. 4)
ENEA-GARR
9 PoP, 18-400 Mbps
Brindisi 150 Mb/s
Bologna 30 Mb/s
Casaccia 100 Mb/s
Frascati
155 Mb/s
Portici 400 Mb/s
Trisaia 18 Mb/s
Palermo
Pisa
Roma Sede
Fig.4 GARR network layoutENEA GRID Web
ACCESS
25
MAGO
Monitoring All Grid Objects
Anna Jannace, Carmine Spizuoco, Francesco Adinolfi(1), Giovanni Bracco (2)
(1)Consorzio
Campano per l’Informatica e l’Automazione Industriale (C.R.I.A.I.),
Piazzale E. Fermi 1, Loc. Porto del Granatello, 80055 Portici, Napoli, Italia.
Phone:+39-081-776-6905, Fax:+39-081-776-0583
[a.jannace, c.spizuoco]@criai.it
(2)ENEA FIM (Servizio Informatica e Reti)
Via E. Fermi 45, I-00044 Frascati (Roma) Italy
Phone: +39-06-9400-5597, Fax: +39-06-9400-5735
bracco@frascati.enea.it
La ricerca è stata effettuata nell’ambito delle attività del SPI.2 del Progetto PON
CRESCO (Centro computazionale di RicErca sui Sistemi COmplessi) di ENEA.
Il progetto ha come obiettivo la realizzazione, presso il Centro Ricerche ENEA di
Portici (NA), di un importante Polo di calcolo multidisciplinare per lo studio dei
sistemi complessi di natura biologica e tecnologica.
26
L’attività SPI-2 del Progetto CRESCO prevede l’implementazione di soluzioni
innovative in tema di architetture di sistemi di calcolo e di GRID computing per le
attività R&S di punta dell’ENEA che richiedano l’utilizzo di risorse computazionali
estremamente importanti.
In questo contesto si inserisce a carico del CRIAI uno studio dei sistemi di
monitoraggio per la realizzazione del sistema informativo delle risorse, secondo lo
standard del GLUE Schema (http://glueschema.forge.cnaf.infn.it/). Tale studio
effettuato sulla base del sistema infrastrutturale ENEA Grid e della sue
caratteristiche ha determinato la scelta di Ganglia (http://ganglia.info/) come punto
d'appoggio del sistema di monitoraggio.
Il presente documento contiene la descrizione del sistema di monitoraggio MAGO
nei suoi componenti WEB, CORE e DB.
Nella precedente figura è rappresentata l’architettura di tale sistema denominato
MAGO (Monitoring All Grid Object). In realtà la figura mette in evidenza moduli
diversi realizzati con diverse tecnologie.
ARCHITETTURA MAGO
Il sistema MAGO è basato su un’architettura distribuita su tre livelli distinti, e
possiede una struttura di tipo gerarchica per permettere una gestione centralizzata
e minimizzare l’intervento sui
singoli nodi.
Una struttura modulare ed
estendibile.
Livello Central Server
Livello Site
Livello Host
27
PROGETTAZIONE
Di seguito viene riportato una vista dei componenti principali del sistema MAGO :
WEB




Configurazione
Visualizzazione degli Allarmi
Inserimento di nuove metriche
Interrogazione del database
28
 AFS
 Contenitore dei sorgenti, applicazione, script
 Memorizzazione delle configurazioni, per ogni sito e host
 Memorizzazione delle Metriche
 CORE
 Prelievo delle metriche rilevate dai demoni di ganglia Gmetad
 Installazione automatica dell’ambiente Mago sugli Host
 Trasferimento delle informazioni sul server di sito
 Prelievo decodifica e inserimento delle metriche
 MAGO.sql
Database di grosse dimensioni, che conserva:
 Informazioni di configurazione
 Metriche inserite
 Misure rilevate
 Configurazioni delle sottoreti
 Allarmi
Inoltre attraverso una User Defined Function segnala all’interfaccia Web
la presenza di metriche ritenute critiche.
Flusso dell’informazione
Tale sistema si poggia sul tool di monitoraggio GANGLIA, utilizzando i demoni
Gmond e Gmetad per la replicazione e il prelievo dell’informazione tra Site e
Host.
Di seguito è riportato il percorso dei dati dopo che sono stati prelevati da Gmetad
di Ganglia.
29
MAGO WEB
La Web Application di MAGO fornisce una interfaccia utente che permette :
 Accesso sicuro con le credenziali standard di ENEA-Grid
 Configurazione dei siti, degli host, delle sottoreti
 Inserimento di nuove metriche
 Visualizzazione dei risultati
30
Fault Tolerance
Tale sistema poggia la sua affidabilità sulla persistenza dei dati, in quanto
l’informazione sensibile è rimossa da ogni sottosistema (Site,Host) se e solo se
esiste una copia sul server centrale.
Al fine di rendere stabile questa operazione, è stato necessario implementare un
modulo di controllo dei demoni dipendenti.
Esso permette di prevenire anomalie nell’informazione dovuti a malfunzionamenti.
31
Inoltre si occupa della riattivazione automatica dei demoni dopo aver rilevato
un’anomalia nel sistema.
NEW TOOLS AND TECHNOLOGIES FOR THE
ENEA-GRID PERVASIVE INTEGRATION
ALESSIO ROCCHI1, GIOVANNI BRACCO1, ANDREA SANTORO1, CARLO SCIÒ2 {ALESSIO.ROCCHI,
BRACCO, ANDREA.SANTORO, SCIO}@ENEA.IT
(1)ENEA – CR FRASCATI - V. ENRICO FERMI 45, FRASCATI (ROMA) – (2)ESSE3ESSE
ABSTRACT
32
The ENEA-Grid environment lies on two fundamental pillars, the Andrew File System
(AFS, , providing facilities for the resource distribution over a WAN), and the Load Sharing
Facility (LSF, able to handle the resource and load management). In this context, the
CRESCO project is intended to be the main activity in progress.
In the current occasion we want to present three innovative projects that have been
engineered within CRESCO.
AMACA (AFS Memorize and Check Application) is a two-module application aiming to
trace the status of every AFS core component, while providing an open, state-of-the-art
monitoring platform for both administrators and users. One of the main AMACA succesful
goals is to expose a set of APIs (already adopted by other supporting tools) for the native
web-based authentication towards AFS.
ARCO (AFS Remote Command Operator) is a Service Oriented web-based application,
whose purpose is to submit remote commands to machines registered by system
administrators.
A special mention is deserved to another solution, formed by a set of open, standard tools
whose interoperability guarantee a native interface between AFS and the Apache web
server.
1. AMACA (AFS MEMORIZE AND CHECK APPLICATION)
AMACA is a two-module application whose purpose is to track issues and working
statuses over the OpenAFS distributed filesystem, while offering a convenient way to the
system administrators in studying and analyzing them.
A simple diagram on how AMACA modules work is represented in the next image 1 .
IMAGE 1 - AMACA MODULES AND INVOLVED COMPONENTS
1.1 CRAWLER
The crawler module of AMACA (based on N. Gruener’s AFS APIs) is a Perl application
aiming to check the status of every AFS core component. The indexing results are handled
by a memorization module, that is responsible to store the history of file system's events in
a MySQL backend (making easier to perform subsequent fine-grained data mining
operations). Every crawler invocation is differentiated by the others by an unique ID called
snapshot. As the crawler can be invoked both directly (i.e.: a user belonging to a certain
PTS group explicitly runs it via the web Explorer module) or not (i.e.: its execution is
scheduled in the O.S.’s cron), also the database is splitted in two: a “scratch” portion is
dedicated to the latter execution kind, so that the administrator can solve problems and
33
see “instantly” what is the effect of his intervention without interfering with the global
informations.
1.2 EXPLORER
The explorer module of AMACA is a web 2.0 compliant application, written in Php with
some AJAX elements, providing a comfortable interface to the analysis of the crawling
results. The Explorer module provides facilities both for the visualization of “static data”
(i.e.: size and number of volumes and partition, sync site, alarms about critical events, …)
and interactive queries. The Explorer module is adaptive: every shown data can be shrunk
to a particular snapshot/domain in order to focus the attention only on situations detected
locally.
2. AUTHENTICATION APIS
AMACA exposes a set a well-structured interface for the user web authentication over
OpenAFS. This API realizes an IPC mechanism with the PAG shell, so that an user can
get access to a protected resource simply providing its ENEA-GRID username and
password, in a convenient, integrated way. The authentication APIs work atomically (The
PAG Shell provides an “user-aware” isolation level for an object -the token- whose
existence, otherwise, would be limited to one instance at a time, generating harmful race
conditions), and securely (because of the presence of two cryptographic layers: SSL and
RSA via mcrypt-lib)
IMAGE 2 - AUTHENTICATION APIS BUSINESS DIAGRAM
3. ARCO (AFS REMOTE COMMAND OPERATOR)
ARCO is an application engineered to execute commands on remote machines. Its original
purpose foresaw to be fully focused on LSF multicluster, but it became soon a software
capable to handle indifferently any remote tool).
34
With ARCO (img. 1), an administrator can register the services he wants to deal with, the
machines where the services are (simply by loading files in LSF/text format, but we plan to
extend the support to other file types), and perform the commands execution in a visual,
convenient, secure way. Every task ARCO performs is logged (img. 2) in a MySQL
backend (where also the associations machine/service are stored).
ARCO uses, on the authentication side, the same APIs than AMACA (extended to have a
better support on more PTS groups checking)
4. APACHE AND AFS INTEGRATION
The work on the integration between the Apache web server and OpenAFS has involved
different tasks, referring to three main scenarios:
1. We want our web server to publish the pages of users, projects and software simply by
linking them into the document root to the AFS shared space where they are stored. This
task is accomplished by a set of scripts that fill up the web root in an incremental way.
35
IMAGE 5 - COMMON SCENARIO
3.
Resources accessible only by selected users or groups (fig. 4). This need can’t
be solved by using the authentication APIs (they always need the presence of an index file
for the cookie processing, while a user likes to simply share a simple list of resources), and
it has to be performed at the Apache level.
IMAGE 6 - SCENARIO WITH PROTECTED RESOURCES
We engineered a patch for the Jan Wolter’s mod_authnz_external apache module (aiming
to bypass the built-in apache authentication mechanism, in order to get it tailored on the
user needs), so that it could export the current URI’s physical path. The Apache
environment is passed to two other software modules, performing the control over the
requestor’s membership and mapping the results on the current directory rights .
IMAGE 7 - WORKING MODEL OF THE ENTIRE SYSTEM
(Partially solved) issue: slightly increasing response time: the interpreter is called at
every step of the directory walking and this results in a little overhead.
3. Security: the “symlink-aware” structure could be harmful: an incautious user could
create links toward sensitive system files without any control. So we had to
36

Force Apache to follow the link only if its owner matches the one of the link’s target
(system files are usually owned by root);

Fill in the passwd file with the AFS user names, so that it could be possible to chown
every linked page to its AFS owner (and allow only root to be accepted when receiving
a remote access request via ssh).
37
Advances in Interoperability between ENEAGRID and gLite-based Infrastructures
A. Santoro, C. Scio*, A. Rocchi, G. Bracco , S. Migliori, A. Quintiliani, S. Podda
ENEA-FIM, ENEA C.R. Frascati, 00044 Frascati (Roma) Italy, (*) Esse3Esse
Summary
The GRID approach has allowed to integrate in a single unified system, namely ENEAGRID, all the high performance computational resources available inside ENEA. The main
software components of the ENEA-GRID infrastructure are the distributed file system
OpenAFS and the multi-site resource manager LSF Multicluster, which constitute the
production ready ENEA-GRID middleware.
In the participation of ENEA in national and European GRID projects, solutions had to be
found to enable the interoperability between ENEA-GRID and other GRID middleware.
Among these gLite is one of the leading solutions, originated in the EGEE/EGEE-II
projects and adopted in other contexts.
The poster presents the ongoing activity to make ENEA-GRID interoperable with the
EGEE grid infrastructure. As the two grids employ different mechanisms for authentication,
scheduling and data sharing, their integration is not straightforward.
However, the ability for ENEA-GRID to receive jobs from gLite infrastructures has already
been established in previous works. The ENEA EGEE site is in production, using the so
called SPAGO technology (Shared Proxy Approach for Grid Objects).
Now we have extended the ENEA-GRID/gLite interoperability mechanism described
above, by adding the ability for ENEA-GRID users to submit jobs to gLite-based grids in a
transparent way.
The same solutions have also been adopted in implementing the interoperability with
another gLite based infrastructure, namely the GRID connecting the computational
resources of four GRID projects in Southern Italy (ENEA CRESCO, SCOPE, CYBERSAR
and PI2S2). CRESCO computation resource are fully integrated into ENEA-GRID and the
solutions found for EGEE have been straightforwardly applied, in a context where each of
the 4 projects retains the capability to run all the main GRID services.
Two EGEE technical notes have been prepared to document the gateway implementation:
●
EGEE Technical Note EGEE-TR-2007-001
"The gateway approach providing EGEE/gLite access to non-standard
architectures" Bracco, G; Migliori, S; Sciò, C. ; Santoro , A.;
http://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2007-001.pdf
●
EGEE Technical Note EGEE-TR-2006-006
"AFS Pool Account Users - GSSKLOG and LCMAPS extension to support AFS
users as EGEE pool account users"
Bracco, G; Giammarino, L; Migliori, S; Sciò, C.;
http://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2006-006.pdf
38
ENEA
[Italian National Agency for New Technologies, Energy and Environment]
12 Research sites and a Central Computer and Network Service (ENEA-INFO) with 6
computer centres managing multi-platform resources for serial & parallel computation
and graphical post processing.
Computational resources:
 Hardware / OS:
 IBM SP - AIX;
 ia64/x86/x86_64 Linux;
 SGI Altix & Onyx;
 Apple cluster;
 Intel Windows servers.

software:
 commercial codes (fluent, ansys, abaqus);
 elaboration environments (Matlab, IDL);
ENEA GRID architecture
ENEA GRID mission [started 1999]:
 provide a unified user environment and an homogeneous access method for all
ENEA researchers, irrespective of their location.
 optimize the utilization of the available etherogeneous resources.
39
GRID functionalities (unique authentication, authorization, resource access and resource
discovery) are provided using “mature”, multi-platform components:
 Distributed File System: OpenAFS
 Resource Manager: LSF Multicluster [www.platform.com]
 Unified user interface: Java & Citrix Technologies
These components constitute the ENEA-GRID Middleware.
OpenAFS
● user homes, software and data distribution
●
integration with LSF
●
user authentication/authorization, Kerberos V
Interoperability gLite vs ENEA-GRID
The gLite software is the GRID middleware developed in the context of the EGEE project.
Its authentication scheme is based on X509 certificates as an authentication mechanism,
its data sharing is enabled through the existence of Storage Elements visible to the whole
GRID, and the scheduling is carried out by a software element known as Workload
Management System (WMS). Conversely, ENEA-GRID employs a middleware based on
the geographically distributed filesystem OpenAFS for data sharing, the resource manager
LSF Multicluster for scheduling, and a combination of Kerberos 5 and AFS tokens as
authentication mechanism. Interoperability between such different systems consists of two
different parts. Submitting jobs from gLite to ENEA-GRID and submitting jobs from ENEAGRID to gLite.
gLite vs ENEA-GRID
Job submission in gLite infrastructure:
edg-job-submit <file.jdl>
<file.jdl> is an ASCII format file containing the information about the
executable file and the files that must be transferred on the WN to allow a
proper job execution
Job submission to ENEA-GRID:
bsub -q <queuename> <file.exe>
<queuename> is the name of the specific queue that must execute the job
<file.exe> is the executable file. Note that all the other required files are
already exported to each machines in ENEA-GRID by AFS, thus they not
need to be notified to bsub.
Note: gLite users need to notify into the jdl all the input/output files required by the job;
conversely ENEA-GRID users have no such requirement, which might lead to inconsistent
communication between the two middlewares. See the issue below: “Transparent File
sharing”.
gLite to ENEA-GRID: The SPAGO approach
The basic design principle of the SPAGO approach, that allows ENEA-GRID to process
jobs submitted to gLite, is outlined in Figure 1 and it exploits the presence of AFS shared
file system. When the CE receives a job from the WMS, the gLite software on the CE
40
employs LSF to schedule jobs for the various Worker Nodes, as in the standard gLite
architecture.
gLite to ENEA-GRID interface: SPAGO Approach
However the worker node is not capable to run the gLite software that recovers the
InputSandbox. To solve this problem the LSF configuration has been modified so that any
attempt to execute gLite software on a Worker Node actually executes the command
on a specific machine, labeled Proxy Worker Node which is able to run standard gLite.
By redirecting the gLite command to the Proxy WN, the command is executed, and the
InputSandbox is downloaded into the working directory of the Proxy WN.
The working directory of each grid user is maintained into AFS, and is shared among all
the Worker Nodes and the Proxy WN, thus downloading a file into the working directory of
the Proxy WN makes it available to all the other Worker Nodes as well. Now the job on the
WN1 can run since its InputSandbox has been correctly downloaded into its working
directory. When the job generates output files the OutputSandbox is sent back to the
WMS storage by using the same method.
In the above architecture, the Proxy WN may become a bottleneck since its task is to
perform requests coming from many Worker Nodes. In that case a pool of Proxy WN can
be allocated to distribute the load equally among them.
ENEA-GRID to gLite
41
Job submission from gLite to ENEA-GRID took advantage of the fact that gLite CE
employs LSF Multiclaster as one of its resource managers. Therefore slight modifications
in the configuration of LSF allows seamless interfacing between the gLite CE and the
underlying ENEA-GRID infrastructure.
On the other hand LSF multicluster does not have embedded supports to interface with
gLite middleware, which leads to a more complex approach. The overall design approach,
shown in Figure 2, is the following: an ENEA-GRID user who wants to submit a job to
gLite, submits its request to LSF (e.g. using “bsub” command), as he would do for any
ENEA-GRID job, but specifies a specific “gLite-interface” queue for the job. In its turn the
LSF queue redirects the job towards a special software module that generates a proper jdl
file and forwards the job and its jdl to a gLite User Interface. From there it is responsibility
of the gLite middleware to send the job to the appropriate Computing Element and report
to the interface software module when the job is completed.
ENEA-GRID to gLite interface
Issues under investigation
The ENEA-GRID to gLite interface presents still two issues to be solved in order to have a
full ENEA-GRID/gLite interoperability:
Transparent Authentication: In order to be able to use the gLite infrastructure the user is
expected to have a personal X509 certificate released by the proper certification
authority. This is a requirement of the EGEE project, and unavoidable. However,
once the user has installed correctly his personal certificate on his machine he should
be able to access the whole gLite infrastructure through the ENEA-GRID interface
described above. Currently this is not the case, since the user must issue a command
“setup-egee-proxy.sh” (analogous to the voms-proxy-init in gLite) to generate a proxy
certificate on the gLite User Interface. Since ENEA-GRID employs its own
credentials, we are studying a mechanism that may automatically translate the
kerberos tickets used by ENEA-GRID into a X509 proxy certificate, thus providing
transparent authentication.
42
Transparent File Sharing: In the ENEA-GRID infrastructure all the machines share the
same filesystem due to the use of OpenAFS. This means that there is no need to
specify the files required by the jobs to run correctly, since the files are shared among
all the machines that might run the job. On the other hand the gLite infrastructure
requires to identify into the jdl file all the files required by the job. Our current
approach to this problem consists in asking the user to identify such files into the
bsub invocation, which would be intercepted by our interface software and a proper
jdl file containing the files would be generated. However, the fact that the user must
be aware of the files needed by the submitted job means that the job submission
process from ENEA-GRID to gLite is not completely transparent. We are currently
investigating a more transparent submission mechanism that would allow even jobs
on gLite WN to transparently import the needed files from AFS.
Interoperabilty of GRID projects
In the context of the PON1575 project, there has been a strong focus on integrating the
four GRID projects in Southern Italy (ENEA CRESCO, SCOPE, CYBERSAR and PI2S2)
into a single, interoperable computational GRID. The platform of choice is the gLite
middleware.
CRESCO computation resources are fully integrated into ENEA-GRID and therefore we
have been able to apply the solution described above to make CRESCO project
interoperable with the other three italian projects. Moreover, each of the 4 projects retains
the capability to run all the main GRID services autonomously, so that a failure on one
project will not disable the operations on others.
CRESCO HPC CENTRE
www.cresco.enea.it
CRESCO (Computational Research Center for Complex Systems) is an ENEA Project, cofunded by the Italian Ministry of University and Research (MUR). The project will be
functionally built around a HPC platform and 3 scientific thematic laboratories: the
Computing Science Laboratory, hosting activities on HW and SW design, GRID
technology and HPC platform management
The HPC system (installation 1Q 2008) will consist of a ~2500 cores (x86_64)
resource (~25 Tflops peak), InfiniBand connected with a 120 TB storage area. The
resource, part of ENEA-GRID, will be made available to EGEE GRID using gLite
middle-ware through the gateway approach.
43
GOC/GSTAT page with ENEA-GRID WN information
The ENEA-INFO site has been certified for the production grid service providing access
both to linux and AIX Worker Nodes.
44