Geodise & GeodiseLAB Simon Cox University of Southampton 21st April 2005 © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Grid Enabled Optimisation and Design Search for Engineering (GEODISE) Simon Cox- Grid/ W3C Technologies SYSTEMS- Engineering Southampton, OxfordBAE and Manchester and High Performance Computing Global Grid Forum Apps Working Group Andy Keane- Director of Rolls Royce/ BAE Systems University Technology Partnership in Design Search and Optimisation Mike Giles- Director of Rolls Royce University Technology Centre for Computational Fluid Dynamics Carole Goble- Ontologies and DARPA Agent Markup Language (DAML) / Ontology Inference Language (OIL) Nigel Shadbolt- Director of Advanced Knowledge Technologies (AKT) IRC Rolls-Royce- Engineering Fluent- Computational Fluid Dynamics Microsoft- Software/ Web Services Intel- Hardware Compusys- Systems Integration Epistemics- Knowledge Technologies Condor- Grid Middleware The GEODISE Team ... • • • • • • • • • • • • • • • Richard Boardman Sergio Campobasso Liming Chen Mike Chrystall Trevor Cooper-Chadwick Simon Cox Mihai Duta Clive Emberey Hakki Eres Matt Fairman Mike Giles Carole Goble Ian Hartney Tracey Hunt Zhuoan Jiao • • • • • • • • • • • • • • • Andy Keane Marc Molinari Graeme Pound Colin Puleston Nicola Reader Angus Roberts Mark Scott Nigel Shadbolt Wenbin Song Paul Smart Barry Tao Lisa Telfer-Brunton Jasmin Wason Fenglian Xu Gang “Luke” Xue © Geodise Project, University of Southampton, 2001-2005 http://www.geodise.org/ Distributed Systems 2005 IP Moore’s Law ASCII HTTP (HTML) XML (HTML) Web Services Network Compute/ Data Future Services Flexible, Seamless, and Secure Sharing © Geodise Project, University of Southampton, 2001-2005 http://www.geodise.org/ Software Applications - Geodise CFD 3D engine nacelle optimisation 0 1 2 Conventional Inlet Negative Scarf Inlet Goal - reduce ground noise generated by fan when plane takes off. Optimise aerodynamic performance when scarf angle is varied. © Geodise Project, University of Southampton, 2001-2005 http://www.geodise.org/ Workflow for aerodynamic shape optimisation using CAD, Gambit, and Fluent Problem definition Design of Experiment ProEngineer CAD (Condor Pool) Gambit Meshing (Globus Compute) Response surface modelling Optimisation on Response surface Validation Fluent CFD (Globus Compute) Applications – GEM Electromagnetic optimisation Transmission of light properties through a photonic crystal Parameters are radius of holes and light wavelength • Large number of designs, parameters and solutions • Query for a particular data range to postprocess © Geodise Project, University of Southampton, 2001- 2004. http://www.geodise.org/ GEODISE Engineer GEODISE PORTAL Reliability Security QoS Visualization Knowledge repository Session database Traceability Ontology for Engineering, Computation, & Optimisation and Design Search OPTIMISATION OPTIONS System APPLICATION SERVICE PROVIDER Intelligent Application Manager CAD System CADDS IDEAS ProE CATIA, ICAD Globus, Condor, OGSA Optimisation archive COMPUTATION Licenses and code Analysis CFD FEM CEM Parallel machines Clusters Internet Resource Providers Pay-per-use Intelligent Resource Provider Geodise will provide grid-based seamless access to an intelligent knowledge Design repository, a state-of-the-art collection of optimisation and search tools, archive industrial strength analysis codes, and distributed computing & data resources / n b n o a o l h t t & h Ma P yt / Jy Grid-Enabled Scripting Environment Collaboration Coordination Applications Enterprise • Why use scripting languages? Grid Management – Flexibility – High-level functionality – Quick application development – Extend the user’s existing PSE – … Acts as execution/ enactment engine too • Approach – Matlab initially chosen as the hosting environment – Now working with Python/ Jython © Geodise Project, University of Southampton, 2001-2005 http://www.geodise.org/ Hardware Engineering Applications of the Grid via Geodise Problem Solving Environment Matlab, Python, etc Bringing underlying infrastructure into the Problem Solving Environment Geodise Toolkits “Useful into useable” “Dial-tone” Enterprise “Heartbeat” Grid Hardware • Grid Compute toolbox • Cycle stealing High-performance computing Integration with heterogeneous environments (Windows, Linux, Solaris, …) Grid Data toolbox Raw files into journaling filesystem Metadata in relational database (Oracle, DB2, SQL Server, mySQL…) XML processing capability • Problem Solving Environments e.g. Matlab, iSight Python, ModelCenter,… Optimisation `Tools Geodise deliverables Intelligent Workflow composition, execution and monitoring tool Rule and case-based composition of ‘best’ workflow (semantics and knowledge driven) Deployment to compute/ data resources System-level “heartbeat” User/ Application level “dial-tone” (security, resource availability, network weather, …) Grid-enabled Toolboxes for Engineers Middleware (Grid) Hardware Designed around web-services to inherit and leverage Technologies (e.g. security, reliability) and Development tools Collaboration Geodise Tools Coordination Applications Enterprise • A number of Grid-enabled design tools have been developed Grid Management – Compute Toolbox: certificate based authentication, job Hardware submission, and file transfer, etc. – Database Toolbox: archive files and data, custom metadata, locate and retrieve data, etc. – XML Toolbox: convert Matlab structures to and from XML – Workflow Editor: graphical problem solving environment which allows users to construct and configure workflows for the Grid – OptionsMatlab: Matlab interface to the Options design exploration system – Knowledge Toolbox: Intelligent Design advice © Geodise Project, University of Southampton, 2001-2005 http://www.geodise.org/ Engineer Graphical Workflow Tool Results Geodise Architecture Results Knowledge Toolbox Compute Toolbox API Results CAD Globus Servers OptionsMatlab Matlab Script Pro/Engineer Globus Server #1 Database Toolbox Compute Toolbox ICAD Grid Job Submission Condor Cluster Geodise Database Globus Server / Condor Node Globus and Condor Job Managers XML Toolbox Results by GridFTP Catia Globus Server #N Metadata by webservice Database Toolbox API Knowledge Toolbox API Applications Computation Matlab Scripting Environment IDEAS Analysis Gambit/Fluent Abaqus Metadata Webservice Condor Master Catia HYDRA Globus Server Knowledge Repository Condor Node #1 Physical files, scripts, and outputs Metadata and Structure Repository Condor Node #N Local Filesystem Java Matlab Sponsors © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ GeodiseLab Activities for the OMII • Harden & deliver Geodise toolboxes – Geodise Compute Toolbox – Geodise Database Toolbox – XML Toolbox for Matlab • • • • • Documentation Testing Extend existing functionality Train and support users ... GGF: Simple API for Grid Applications (SAGA) Working Group © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Compute Toolbox: Aims • Support access to whichever compute resources users wish to exploit – Globus GT2 – OMII services – Condor • Provide access from alternative environments – Matlab – Jython – cPython • Leverage existing APIs wherever possible © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Compute Toolbox: Hit-list Matlab Jython cPython .m files .py files .py files ? Geodise Java API .NET Client Windows HPC/Grid Condor Native Java Client Java CoG OMII API Condor Web Service Globus GT2 OMII Services © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Available Now GeodiseLab Compute Toolbox © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Compute Toolbox Release Matlab Jython cPython .m files .py files .py files ? Geodise Java API Condor Native Java Client Java CoG OMII API Condor Web Service Globus GT2 OMII Services © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Toolbox Architecture Matlab Environment script.m gd_jobsubmit.m gd_getfile.m Java CoG Globus GT2 Resource GRAM job submission GridFTP file transfer GRAM Service GridFTP Service © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Certificate Management Functions gd_certinfo Returns information about the user's certificate gd_createproxy Creates a Globus proxy certificate gd_proxyinfo Returns information about the user's proxy certificate gd_proxyquery Queries whether a valid proxy certificate exists gd_destroyproxy Destroys the local copy of the user's Globus proxy certificate © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Certificate Management C:\GEODISE >> gd_createproxy © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Certificate Management C:\GEODISE >> subject = gd_certinfo subject : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound issuer : C=UK,O=eScience,OU=Authority,CN=CA,E=ca-operator@gridsupport.ac.uk start date : Wed Sep 22 16:31:24 BST 2004 end date : Thu Sep 22 16:31:24 BST 2005 subject = /C=UK/O=eScience/OU=Southampton/L=SeSC/CN=graeme pound >> isvalid subject : issuer : type : strength : timeleft : = gd_proxyinfo C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound,CN=proxy C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound full legacy globus proxy 512 bits 11 h, 59 min, 30 sec isvalid = 1 >> © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Job Submission Functions gd_jobstatus Gets the status of a Globus GRAM job gd_jobsubmit Submits a compute job to a Globus GRAM job manager gd_jobpoll Queries the status of a Globus GRAM job until complete gd_jobkill Kills a Globus GRAM specified by a job handle gd_chmod Changes file permissions of a file on a Globus resource gd_condorsubmit Submits a job to a Condor pool via a Globus resource gd_submitunique Submits a GRAM job to a unique working directory © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Job Submission C:\GEODISE >> RSL = '&(executable="/bin/date")(stdout="date.out")'; >> host = 'grid-compute.oesc.ox.ac.uk'; >> jobhandle = gd_jobsubmit(RSL, host) jobhandle = https://grid-compute.oesc.ox.ac.uk:30001/30705/1098694366/ >> isdone = gd_jobpoll(jobhandle) isdone = 1 >> © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ GridFTP File Transfer gd_getfile Retrieves a remote file using GridFTP gd_putfile Puts a remote file using GridFTP gd_transferfile Performs a third-party file transfer using GridFTP gd_makedir Creates a remote directory using GridFTP gd_listdir Lists the contents of a directory on a GridFTP resource gd_fileexists Tests the existence of files on a Globus resource gd_rmdir Deletes a remote directory using GridFTP gd_rmfile Deletes a remote file using GridFTP gd_rmuniquedir Deletes a remote directory and its contents © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ File Transfer C:\GEODISE >> exists = gd_fileexists(host, 'date.out') exists = 1 >> gd_getfile(host,'date.out','localfile.txt'); >> type('localfile.txt') Mon Oct 25 09:52:46 BST 2004 >> gd_rmfile(host, 'date.out'); >> exists = gd_fileexists(host, 'date.out') exists = 0 >> © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Resource Testing gd_servermetrics Performs tests upon a list of Globus resources gd_testauthentication Tests authentication with a Globus resource gd_testfiletransfer Tests file transfer to a Globus resource gd_testjobsubmission Tests the job submission to a Globus resource gd_timeauthentication Times authentication to a Globus resource gd_timefiletransfer Times file transfer to a Globus resource gd_timejobsubmission Times a job submission to a Globus resource © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Resource Testing C:\GEODISE >> >> >> >> servers(1).name = 'grid-compute.leeds.ac.uk'; servers(2).name = 'escience-dept2.sesnet.soton.ac.uk'; results = gd_servermetrics(servers); disp(results(1)) name: 'grid-compute.leeds.ac.uk' authentication: 188 jobsubmission: 672 filetransfer: 984 >> disp(results(2)) name: authentication: jobsubmission: filetransfer: 'escience-dept2.sesnet.soton.ac.uk' 125 485 4938 >> © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Available Now GeodiseLab XML Toolbox © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Web Technology & Services ID Name Value 0001 A 1 0002 B 2 0003 C 3 Databases Access Engineer’s familiar scripting environment XML Toolbox Communication © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Features Toolbox <struct xmlns="http://www.geodise.org/matlab.xsd" idx="0" fields="name id param"> <char idx="1" name="name" size="1 14">eng_opt_design</char> <char idx="1" name="id" size="1 11">1022223-779</char> <struct idx="1" name="param" size="1 1" fields="width height material"> <double idx="1" name="width" size="1 1"> 100 </double> <double idx="1" name="height" size="1 1"> 87.9 </double> <double idx="1" name="material" size="1 1"> 0.07244 </double> </struct> </struct> Filestore Toolbox • • • • • • DB Standard File Formats (eg. STEP-XML, …) <root> <name>eng_opt_design</name> <id>1022223-779</id> <param> <width> 100 </width> <height> 87.9 </height> <material> 0.07244 </material> </param> </root> Powerful Matlab to XML interface Allows transparent conversion between proprietary Matlab data types and standard XML Brings flexibility of open XML Standard to Matlab Easy-to-use interface (4 simple core functions: xml_save, xml_load, xml_format, xml_parse) Extensive documentation & direct help functionality Large range of potential applications: – Science, Engineering, IT, Web, … © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ XML Toolbox in action • Jan 2004 v2.0 released • Nov 2004 – Downloads from soton.ac.uk/~gridem: 227 – Downloads from mathworks.com/matlabcentral: 882 • Apr 2005 v3.0 – > 1500 downloads total • Large user base – Academic & Commercial – Extensive testing & feedback community – Market pull • So far very positive feedback from users • Matlab Central user rating: 5 / 5 • Linked by Mathworks help for XML in Matlab © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Aalesund College: Read data from database Aberdeen University: Export analysis results from Matlab to XML ADVA: to read existing XML-files in a Matlab structure and do some calculations Advanced Tech Center: configuring xml input into matlab Alfred-Wegener-Institut: Handle data from echo sounder AMAT: Evaluation of XML input in matlab Anna University: We are programmed to develop a new tool for analysing discrete event system using matlab. Applied Math Labolatory, Ecole Centrale Paris Aston University: To read date from xml files in matlab Avantium Technologies BV: Transferring HTS data stored as XML into Matlab BAE Systems: Data import/export between MATLAB and Delphi; Generation of XML files to control a BAE SYSTEMS Software Defined Radio Beb Corp Bosch GmbH Calypso Medical: looking at xml files in matlab Carnegie Mellon university: Use XML in our facial expression analysis project Center for Advanced Spatial Technologies: Writing least squares adjustments for geodetic observations as part of research in new techniques for parameter estimation combining photogrammetric and terrestrial observations. Data is varied but structured and I am interested in using XML Chouaib Doukkali Universit: Numericals simulations of the physics phenomenons Christian-Albrechts-Universitat zu Kiel (CAU): http://busselab.uni-kiel.de City College of CUNY: Projects using Matlab and XML technology. CMS: For trial Cold Spring Harbor Laboratory: Storing and parsing neuronal data Conestoga college: intro to telecomme Crystal Engineering CU: Sending sending commands to matlab via xml, outputting results of calculations from matlab in xml to be used by other products Daimler Chrysler: test DeAM: xml reading DECO DEIMOS Engenharia: Reading/writing configuration files DGA Defence, France: For managing data files used in simulation's input DSTO: input parameters to radar model EADS CRC: Information Emyo: For teaching ENI: Reading & Modifying XML documents EPFL: Evaluation for data exchange between matlab and other software Fairchild Dornier: Currently I make a research, how I can efficiently store my matlab data structure in a text editable data format, that's why I'am looking for a xmlread and a xmlwrite function. Financial Engines FIOT © Geodise Project, University of Southampton, 2001-2005. Fraunhofer Germany: The last aim of our research is to provide user a platform for maintenance oriented decision via web service. 38 sample institutions http://www.geodise.org/ © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Available Now GeodiseLab Database Toolbox © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Database Toolbox – Aims • Make data easier to manage, find and share • Store data with descriptive information – Standard and user defined metadata – Query over metadata to easily locate data • Familiar interface for engineers – Wrap toolbox as Matlab/Jython functions – Easy integration into user scripts © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Database Toolbox – Features • Archive files and data structures with metadata • Group related data • Simple query interface • Retrieve data based on ID rather than location • Security – User grants permission for others to access their data – Certificate-based authentication and authorisation – Secure Web service access to central database © Geodise Project, University of Southampton, 2001-2005. http://www.geodise.org/ Functions Overview gd_archive Stores a file or structure with metadata into an archive gd_query Queries over metadata or structures in the archive gd_retrieve Retrieves a file or structure from the archive to the local machine gd_addusers Grants users permission to access some data gd_datagroup Creates a new datagroup to aggregate files and structures gd_datagroupadd Adds a file or structure to a datagroup gd_display Displays the results of a query © Geodise Project, University of Southampton, 2005. http://www.geodise.org/ GeodiseLab Database Toolbox Worked Example © Geodise Project, University of Southampton, 2005. http://www.geodise.org/ Client Generate Data Grid Location Service Metadata Service Authorisation Service Query Service XML Toolbox Java CoG Globus Server Database Client Metadata Grid Location Service Metadata Service Authorisation Service Query Service XML Toolbox Java CoG Globus Server Database Client Archive Grid Location Service Metadata Service Authorisation Service Query Service XML Toolbox Java CoG Globus Server Database Client Generate More Data Grid Location Service Metadata Service Authorisation Service Query Service XML Toolbox Java CoG Globus Server Database Client Group Files Grid Location Service Metadata Service Authorisation Service Query Service XML Toolbox Java CoG Globus Server Database Client Query Grid Location Service Metadata Service Authorisation Service Query Service XML Toolbox Java CoG Globus Server Database Client Retrieve Grid Location Service Metadata Service Authorisation Service Query Service XML Toolbox Java CoG Globus Server Database Applications - GENIE Earth system model tuning studies GENIE supports environmental scientists modelling long term climate change. Optimising a parameterised model over a multi-dimensional state space. Database used for monitoring, sharing, post-processing and fault tolerance. 6 8 x 10 90 (b) 7 0.68 0.66 0.68 60 0.7 Latitude [°N] Atmospheric Diffusivity, m2s−1 7.5 0.64 6.5 6 5.5 5 30 0 −30 −60 4.5 −90 −260 −230 −200 −170 −140 −110 −80 −50 −20 Longitude [°E] 4 10 40 70 100 3.5 3 150 450 400 350 300 250 200 Atmoshperic CO concentration, ppm 2 500 0 5 10 15 © Geodise Project, University of Southampton, 2001-2005 http://www.geodise.org/ 20 25 30 Distributed Systems: 2005 and beyond? IP Moore’s Law ASCII HTTP (HTML) XML (HTML) Web Services Network Compute/ Data Software Services Platform Strategies Development Tools Systems Architecture Distributed Data Management Problem Solving Environments Algorithms, Methods and Libraries Heterogeneous Distributed Computing Robustness Security Reliability Visualization Quality of Service Knowledge Technologies Collaboration tools & frameworks Experiences © Geodise Project, University of Southampton, 2005. http://www.geodise.org/