Enabling Grids for E-sciencE EGEE and the European strategy for Grids Mike Mineter NeSC Edinburgh mjm@nesc.ac.uk www.eu-egee.org INFSO-RI-508833 Acknowledgements Enabling Grids for E-sciencE This presentation includes slides and information from: – Fabrizio Gagliardi and Bob Jones (UK AHM 2004 talk) – Roberto Barbera (Slides on applications) – Other colleagues in EGEE • Additional slides and preparation by Mike Mineter, NeSC INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 2 Contents Enabling Grids for E-sciencE • The EU strategy for grids • EGEE – Project goals – Project Activities Operations Middleware Re-engineering (Human) Networking • FAQ about EGEE INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 3 EU strategy -1 Enabling Grids for E-sciencE • From 2002 to 2006, EU funding for Grid research and deployment more than doubled, reaching 275 M€ in FP6. • For the same period, an estimate of the funding for Grid research and deployment by a number of Member and Accession States (UK, France, Italy, The Netherlands, Germany, Hungary, Spain, Poland, Czech Republic, Sweden) is about 300-500 M€. • http://www.gridcoord.org/grid/portal INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 4 EU strategy -2 Enabling Grids for E-sciencE • Romano Prodi, President of the European Commission (quoted by Oluf Nielsen, European Commission ) • “the story of the e-Economy is complex, but it is one we need to understand since the prosperity of the EU Citizens depends on it” • “Education, training and research are the key to economic renewal… we need an integrated strategy for education and research based on networking and mobility giving priority to the technologies of the future” INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 5 EGEE Area in FP6 structure Enabling Grids for E-sciencE EGEE Area INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 6 Advanced Grid Technologies, Systems and Services Enabling Grids for E-sciencE • Information Society Technologies • http://www.cordis.lu/ist/grids/index.htm • Objectives – To advance the current generation of Grids towards the knowledge Grid and complete virtualisation of Grid resources. To foster uptake and use in business and society. – To reduce the complexity of Grid-based systems, empowering individuals and organisations to create, provide access to and use a variety of services, anywhere, anytime, in a transparent and cost-effective way, realising the vision of a knowledge-based and ubiquitous utility. INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 7 Contents Enabling Grids for E-sciencE • The EU strategy for grids • EGEE – Project goals – Project Activities Operations Middleware Re-engineering (Human) Networking • FAQ about EGEE INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 9 EGEE – towards e-infrastructure Enabling Grids for E-sciencE EGEE will build a large-scale production grid service to: • Underpin research, technology and public service • Link with and build on national, regional and international initiatives • Foster international cooperation both in the creation and the use of the einfrastructure INFSO-RI-508833 Pan-European Grid Operations, Support and training Collaboration Network infrastructure & Resource centres EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 10 EGEE has a scope much wider than… Enabling Grids for E-sciencE • Europe: it is an International project with partners world-wide, and funding from the EU • e-Science: It is intended to also support non-scientific research and collaborations in industry, the public sector, … (health, virtual digital libraries…) • Its end date of March 2006: the goal of EGEE is to create an infrastructure that will be sustainable, far beyond the end of its initial phase of funding. (LHC comes on line in 2007) INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 11 Grids for eInfrastructure… Enabling Grids for E-sciencE • What is missing? – Production-quality (stable, mature) Grid middleware – Production-quality operational support Grid Operation Centres, Helpdesks, etc. – Multi-discipline grid-enabled application environment Now led by HEP, Bio-info – Administrative and policy decision framework in order to share resources at pan-European scale (and beyond) Areas such as AAA (Authentication, Authorisation, Accounting) End-to-end issues (Network related) Funding Policies (Grid economics) Resource Sharing Policies Usage Policies • EGEE project will tackle most of the above issues INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 12 In 2 years EGEE will: Enabling Grids for E-sciencE • Establish production quality sustained Grid services – 3000 users from at least 5 disciplines – over 8,000 CPU's, 50 sites – over 5 Petabytes (1015) storage • Demonstrate a viable general process to bring other scientific communities on board • Propose a second phase in mid 2005 to take over EGEE in early 2006 INFSO-RI-508833 Pilot New EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 13 In 2 years EGEE will: Enabling Grids for E-sciencE • Establish production quality sustained Grid services – Reliable and secure – 24 hr/day; 7 day/week – Capable of being sustained: ~20 years • Demonstrate a viable general process to bring other scientific communities on board • Propose a second phase in mid 2005 to take over EGEE in early 2006 INFSO-RI-508833 Pilot New EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 14 EGEE Figures & Organization Enabling Grids for E-sciencE • Coordinator: European Organization for Nuclear Research - CERN • 70 leading institutions in 27 countries, federated in regional Grids • 32 M € EU funding in 2004-2005 (twice from partners) INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 15 Contents Enabling Grids for E-sciencE • The EU strategy for grids • EGEE – Project goals – Project Activities Operations Middleware Re-engineering (Human) Networking • FAQ about EGEE INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 16 EGEE Activities Enabling Grids for E-sciencE 32 Million Euros EU funding over 2 years starting 1st April 2004 • 48 % service activities (Grid Operations, Support and Management, Network Resource Provision) • 24 % middleware re-engineering (Quality Assurance, Security, Network Services Development) • 28 % networking (Management, Dissemination and Outreach, User Training and Education, Application Identification and Support, Policy and International Cooperation) INFSO-RI-508833 Emphasis in EGEE is on operating a production grid and supporting the end-users EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 17 Contents Enabling Grids for E-sciencE • The EU strategy for grids • EGEE – Project goals – Project Activities Operations Middleware Re-engineering (Human) Networking • FAQ about EGEE INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 18 EGEE Service Activities Enabling Grids for E-sciencE • Create, operate, support and manage a production quality infrastructure • Offered services: – Middleware deployment and installation – Software and documentation repository – Grid monitoring and problem tracking – Bug reporting and knowledge database – VO services – Grid management services INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 19 EGEE Service Activities Enabling Grids for E-sciencE • Create, operate, support and manage a production quality infrastructure • Offered services: – Middleware deployment and installation – Software and documentation repository – Grid monitoring and problem tracking – Bug reporting and knowledge database – VO services – Grid management services INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 20 Contents Enabling Grids for E-sciencE • The EU strategy for grids • EGEE – Project goals – Project Activities Operations Middleware Re-engineering (Human) Networking • FAQ about EGEE INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 21 gLite Enabling Grids for E-sciencE • “gLite” - the new EGEE middleware (under test) • Service oriented - components that are : – Loosely coupled (by messages) – Accessible across network; modular and self-contained; clean modes of failure – So can change implementation without changing interfaces – Can be developed in anticipation of new uses – … and are (will be) based on (emerging) standards. • Opens EGEE to: – New middleware (plethora of tools now available) – Heterogeneous resources (storage, computation…) – Interact with other Grids (international, regional, national and thematic) • Goal: to outperform and then replace LCG middleware INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 22 Contents Enabling Grids for E-sciencE • The EU strategy for grids • EGEE – Project goals – Project Activities Operations Middleware Re-engineering (Human) Networking • FAQ about EGEE INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 24 The networking activities Enabling Grids for E-sciencE • What are the Networking Activities? • Their goal: Building new and effective VO’s (user communities) • The Training activities • Application support INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 25 (Human) Networking Activities Enabling Grids for E-sciencE • Dissemination and Outreach: 5% of EGEE budget – Dissemination – to actively promote and raise awareness of the EGEE project – Outreach – to identify and contact potential new user communities • Training and Induction: 4% of EGEE budget – Induction – to introduce and orient - users and members – Training – to create, collate, make available and deliver material and courses • Application Identification and Support – Process for selecting new application areas – Supports selected VO’s in porting applications • International cooperation INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 26 Gaining new and effective users Enabling Grids for E-sciencE The “virtuous circle” INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 27 Enabling Grids for E-sciencE Dissemination • 1st project conference, Cork, April • 2nd conference in The Hague – 22-26 November – http://public.euegee.org/conferences/2nd – Over 300 delegates • Websites, Brochures and press releases – For project and general public www.eu-egee.org – Information packs for the general public, press and industry INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 29 User training and induction Enabling Grids for E-sciencE • Training material and courses from introductory to advanced level • Train a wide variety of users both internal to the EGEE consortium and external groups from across Europe • Experience with GENIUS portal and GILDA testbed • Courses inline with the needs of the projects and applications • See http://egee.nesc.ac.uk/ INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 30 Training and Induction Enabling Grids for E-sciencE NeSC Edinburgh UK & Ireland IHEP IMPB RAS ITEP JINR Protvino Russia Moscow Russia Moscow Russia Dubna Russia KU-NATFAK PNPI RRCKI Copenhagen Denmark Petersburgh Russia Moscow Russia GUP Linz Austria FZK Karlsruhe Germany Innsbruck Austria GRNET Athens Greece INFN CESNET Rome Italy Prague Czech Rep. INFSO-RI-508833 BUTE Budapest Hungary II-SAS Bratislava Slovakia ICM PSNC ICI Warsaw Poland Poznan Poland Bucharest Romania ELUB Budapest Hungary MTA SZTAKI TAU Budapest Hungary Tel Aviv Isreal EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 31 Training Courses and Activities Enabling Grids for E-sciencE Develop NA3 processes; requirements analysis Induction for project members Further courses for project members Induction for users Further user and app. dev. training Advanced courses Workshops: EGEE advances, new VO’s Build repository of training material 6 INFSO-RI-508833 12 18 Project month EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 32 Community support Enabling Grids for E-sciencE • The EGEE training material is being made available to the whole community. • A training materials repository allowing flexible searches of the available material is currently under development and testing. – Training activity homepage: http://egee.nesc.ac.uk/ – Roadmap: training events since the start of EGEE http://www.egee.nesc.ac.uk/schedreg/index.html INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 33 Contents Enabling Grids for E-sciencE • The EU strategy for grids • EGEE – Project goals – Project Activities Operations Middleware Re-engineering (Human) Networking • Application support • FAQ about EGEE INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 34 EGEE pilot application: BioMedical Enabling Grids for E-sciencE • BioMedical – Bioinformatics (gene/proteome databases distributions) – Interactive application (human supervision or simulation) – Security/privacy constraints Heterogeneous data formats - Frequent data updates - Complex data sets Long term archiving • BioMed applications deployed – GATE - Geant4 Application for Tomographic Emission – GPS@ - genomic web portal – CDSS - Clinical Decision Support System http://egee-na4.ct.infn.it/biomed/applications.html INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 35 BLAST – comparing DNA or protein sequences Enabling Grids for E-sciencE • BLAST is the first step for analysing new sequences: to compare DNA or protein sequences to other ones stored in personal or public databases. • Ideal as a grid application – trivial to parallelise as independent concurrent jobs on one or more CEs. – Requires resources to store databases and run algorithms – Large user community INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 36 Enabling Grids for E-sciencE Input file Seq1 > dcscdssdcsdcdsc Computing element dedzedzd zedezdze dedzedzd cdscsdcsc zedezdze dedzedzd dssdcsdc cdscsdcsc zedezdze dedzedzd dscbscds dssdcsdc cdscsdcsc zedezdze bcbjbf dedzedzd dscbscds dssdcsdc cdscsdcsc zedezdze bcbjbf dedzedzd dscbscds dssdcsdc cdscsdcsc zedezdze bcbjbf dedzedzd dscbscds dssdcsdc cdscsdcsc Seq1 zedezdze> bcbjbf dscbscds dssdcsdc dedzedzdzedezdze cdscsdcsc bcbjbf dscbscds cdscsdcscdssdcsdc dssdcsdc bcbjbf dscbscdsbcbjbdfn dscbscds dfjvbndfbnbnfbjn bcbjbf bjxbnxbjk:nxbf bscdsbcbjbfvbfvbvfbvbvbhvbhs vbhdvbhfdbvfd bhvdsvbhvbhdvrefghefgdscgdfg csdycgdkcsqkc … Seqn > bvdfvfdvhbdfvb bhvdsvbhvbhdvrefghefgdscgdfg csdycgdkcsqkchdsqhfduhdhdhq edezhhezldhezhfehflezfzejfv dedzedz dzedezd dedzedz zecdscsd dzedezd dedzedz cscdssdc zecdscsd dzedezd dedzedz sdcdscbs cscdssdc zecdscsd dzedezd cdsbcbjb dedzedz sdcdscbs cscdssdc zecdscsd f cdsbcbjb dzedezd dedzedz sdcdscbs cscdssdc zecdscsd f cdsbcbjb dzedezd dedzedz sdcdscbs cscdssdc zecdscsd f cdsbcbjb dzedezd dedzedz sdcdscbs cscdssdc zecdscsd f cdsbcbjb dzedezd sdcdscbs cscdssdc zecdscsd f cdsbcbjb sdcdscbs cscdssdc f cdsbcbjb sdcdscbs f cdsbcbjb f BLAST UI Seq2 > bvdfvfdvhbdfvb DB dedzedzd zedezdze dedzedzd cdscsdcsc zedezdze dedzedzd dssdcsdc cdscsdcsc Seq2 zedezdze> dscbscds dssdcsdc dedzedzdzedezdze cdscsdcsc bcbjbf dscbscds cdscsdcscdssdcsdc dssdcsdc bcbjbf dscbscdsbcbjbdfn dscbscds dfjvbndfbnbnfbjn bcbjbf bjxbnxbjk:nxbf dedzedzd Seqn zedezdze> dedzedzdzedezdze cdscsdcsc cdscsdcscdssdcsdc dssdcsdc dscbscdsbcbjbdfn dscbscds dfjvbndfbnbnfbjn bcbjbf bjxbnxbjk:nxbf BLAST gridification dedzedzdzedezdzecdscsdcscdssdcsd cdscbscdsbcbjbfvbfvbvfbvbvbhvbh svbhdvbhfdbvfdbvdfvfdvhbdfvbhd bhvdsvbhvbhdvrefghefgdscgdfgcsd ycgdkcsqkcqhdsqhfduhdhdhqedezh dhezldhezhfehflezfzeflehfhezfhehf ezhflezhflhfhfelhfehflzlhfzdjazslzd hfhfdfezhfehfizhflqfhduhsdslchlkc hudcscscdscdscdscsddzdzeqvnvqvn q! Vqlvkndlkvnldwdfbwdfbdbd wdfbfbndblnblkdnblkdbdfbwfdbfn INFSO-RI-508833 DB dedzedzd zedezdze dedzedzd cdscsdcsc zedezdze dedzedzd dssdcsdc cdscsdcsc zedezdze dedzedzd dscbscds dssdcsdc cdscsdcsc zedezdze bcbjbf dscbscds dssdcsdc cdscsdcsc bcbjbf dscbscds dssdcsdc bcbjbf dscbscds dedzedzd zedezdze dedzedzd cdscsdcsc zedezdze dedzedzd dssdcsdc cdscsdcsc zedezdze dedzedzd dscbscds dssdcsdc cdscsdcsc zedezdze bcbjbf dedzedzd dscbscds dssdcsdc cdscsdcsc zedezdze bcbjbf dedzedzd dscbscds dssdcsdc cdscsdcsc zedezdze bcbjbf dedzedzd dscbscds dssdcsdc cdscsdcsc zedezdze bcbjbf dedzedzd dscbscds dssdcsdc cdscsdcsc zedezdze bcbjbf dscbscds dssdcsdc cdscsdcsc bcbjbf dscbscds dssdcsdc bcbjbf dscbscds bcbjbf bcbjbf BLAST DB dedzedzd zedezdze dedzedzd cdscsdcsc zedezdze dedzedzd dssdcsdc cdscsdcsc zedezdze dedzedzd dscbscds dssdcsdc cdscsdcsc zedezdze bcbjbf dscbscds dssdcsdc cdscsdcsc bcbjbf dscbscds dssdcsdc bcbjbf dscbscds RESULT BLAST bcbjbf dedzedzd zedezdze dedzedzd cdscsdcsc zedezdze dssdcsdc cdscsdcsc dscbscds dssdcsdc bcbjbf dscbscds bcbjbf BLAST dedzedzd zedezdze dedzedzd cdscsdcsc zedezdze dssdcsdc cdscsdcsc dscbscds dssdcsdc bcbjbf dscbscds DB bcbjbf Computing element EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 37 Earth Observation: Ozone Enabling Grids for E-sciencE • Building on European Datagrid experience • To produce and store the Ozone profiles or columns – Enhance availability • To extend the processing capabilities – Validation against other data – Mid-latitude ozone studies – ... • To facilitate collaboration – Including with emerging large scale European projects GOME instrument (~75 GB - ~5000 orbits/y) ~28000 profiles/day INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 42 Geophysics Applications Enabling Grids for E-sciencE Seismic processing Generic Platform: - Based on Geocluster, an industrial application – to be a starter of the core member VO. - Include several standard tools for signal processing, simulation and inversion. - Opened: any user can write new algorithms in new modules (shared or not) - Free for academic research -Controlled by license keys (opportunity to explore license issue at a grid level) - initial partners F, CH, UK, Russia, Norway INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 44 Computational Chemistry: molecular simulator Enabling Grids for E-sciencE SURFACE Construction of the Potential Energy Surface Ar - Benzene DYNAMICS Dynamical properties Calculation PROPERTIES Calculation of Averaged quantities no Good Results? yes end INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 45 The MAGIC telescope Enabling Grids for E-sciencE • Largest Imaging Air Cherenkov Telescope (17 m mirror dish) • Located on Canary Island La Palma (@ 2200 m asl) • Lowest energy threshold ever obtained with a Cherenkov telescope Aim: detect –ray sources in the unexplored energy range: 30 (10)-> 300 GeV INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 46 The MAGIC Physics Program Enabling Grids for E-sciencE Pulsars AGNs Origin of Cosmic Rays SNRs INFSO-RI-508833 Cosmological -Ray Horizon Tests of Quantum Gravity effects GRBs Cold Dark Matter EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 47 Contents Enabling Grids for E-sciencE • The EU strategy for grids • EGEE – Project goals – Project Activities Operations Middleware Re-engineering (Human) Networking • FAQ about EGEE INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 50 Who else can benefit from EGEE? Enabling Grids for E-sciencE • EGEE Generic Applications Advisory Panel: – For new applications • EU projects: Diligent, SEE-GRID … • Expression of interest: Planck/Gaia (astroparticle), SimDat (drug discovery) http://agenda.cern.ch/age?a042351 INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 51 Links to industry? Enabling Grids for E-sciencE • EGEE Industry Forum – raise awareness of the project in industry to encourage industrial participation in the project – foster direct contact of the project partners with industry – ensure that the project can benefit from practical experience of industrial applications • For more info: http://public.eu-egee.org/industry/ INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 52 Private or Federated Resources? Enabling Grids for E-sciencE For applications that must operate in a closed environment, EGEE middleware can be downloaded and installed on closed infrastructures Approach being used by MammoGrid EGEE sites are administered/owned by different organisations Sites have ultimate control over how their resources are used Limiting the demands of your application will make it acceptable to more sites and hence make more resources available to you INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 53 Intellectual Property Enabling Grids for E-sciencE • The existing EGEE grid middleware (LCG2) is distributed under an Open Source License developed by EU DataGrid – Derived from modified BSD - no restriction on usage (academic or commercial) beyond acknowledgement – Same approach for new middleware (gLite) • Application software maintains its own licensing scheme – Sites must obtain appropriate licenses before installation INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 54 To read more about EGEE… Enabling Grids for E-sciencE • Explore the web site! • www.eu-egee.org INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 55 Summary Enabling Grids for E-sciencE • EGEE is the first attempt to build a worldwide Grid infrastructure for data intensive applications from many scientific domains • A large-scale production grid service is already deployed and being used for HEP and BioMed applications with new applications being ported • Resources & user groups will rapidly expand during the project • A process is in place for migrating new applications to the EGEE infrastructure • A training programme is underway • Prototype “next generation” middleware is being tested (gLite) • Plans for a follow-on project are being discussed INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 56 Further Information Enabling Grids for E-sciencE EGEE www.eu-egee.org LCG lcg.web.cern.ch/LCG/ NeSC www.nesc.ac.uk The Grid Cafe www.gridcafe.org •More EU sites: •http://www.cordis.lu/ist/grids/fp6_grid_projects.htm •http://www.gridstart.org/concertation_mtg.shtml •“Concertation event” and EGEE conference http://public.eu-egee.org/conferences/2nd/programme/outline.html •e-Infrastructures Reflection Group http://www.e-irg.org INFSO-RI-508833 EGEE and e-Infrastructure EGEE Induction, 8 December 2004, NeSC 57