Current Grids Dave Berry Research Manager, NeSC EGEE is funded by the European Union under contract IST-2003-508833 BNCOD: Current Grids – July 6th, 2004 - 1 Acknowledgements • This talk includes slides from previous tutorials and talks delivered by: • • • • the National e-Science Centre the Condor team the Globus Alliance Andrew Grimshaw (U. of Virginia) • Prepared by Dave Berry, NeSC BNCOD: Current Grids – July 6th, 2004 - 2 1 Overview • • • • • • • • The Grid metaphor Cycle stealing Grids Cluster management Data Grids Metacomputing Grids Convergence and co-operation Portals A sample infrastructure BNCOD: Current Grids – July 6th, 2004 - 3 The Grid Metaphor Mobile Access G R I D Workstation M I D D L E W A R E Supercomputer, PC-Cluster Data-storage, Sensors, Experiments Visualising Internet, networks BNCOD: Current Grids – July 6th, 2004 - 4 2 Cycle stealing Grids • Use idle CPU cycles for productive work • This slide shows a Condor system BNCOD: Current Grids – July 6th, 2004 - 5 A popular example: SETI@Home Collect data Find candidate signals Check data integrity Remove Radio Interference 1997: Entropia 1999: United Devices Identify Final Candidates BNCOD: Current Grids – July 6th, 2004 - 6 3 Cluster management • Cluster: off-the-shelf processors linked to provide a high-capacity computing resource • Cluster management: scheduling jobs onto free processors • • Some similarities to cycle stealing Some solutions based on Condor • Example systems • • • • • Platform LSF NASA/Veridian PBS Sun Grid Engine IBM LoadLeveller Nimrod BNCOD: Current Grids – July 6th, 2004 - 7 Data Grids Data Grid Capabilities Federates multiple data sources Provides global naming Works with local and virtual file systems – NFS, XFS, CIFS Accesses data in DAS, NAS, SAN Uses standard interfaces Caches data locally Server Data Partner Application Users Applications Legion G R I D Wide-area access to data at its source location based on business policies, eliminating manual copying and errors caused by accessing out-of-date copies Server Data Department A Desktop Server Data Department B Cluster Application Vendor BNCOD: Current Grids – July 6th, 2004 - 8 4 Metacomputing Grids Site Resources 26 HPSS 4 Site Resources HPSS 24 External Networks 8 Caltech SDSC 4.1 TF 225 TB HPSS 5 Argonne External Networks External Networks Site Resources External Networks NCSA/PACI 8 TF 240 TB Site Resources UniTree BNCOD: Current Grids – July 6th, 2004 - 9 1998: “The Grid” • Various Toolkits • • • Distribution Various Protocols FTP • Security • Single Sign on • Resource Sharing • • • Discovery Process Creation Scheduling • Portability • APIs • Government Agency Buy in BNCOD: Current Grids – July 6th, 2004 - 10 5 The Globus Toolkit (v2) • Grid Security Infrastructure (GSI) • X.509 authentication with delegates and single sign-on • Grid Resource Allocation Mgmt (GRAM) • Remote allocation, reservation, monitoring, control of compute resources • GridFTP protocol (FTP extensions) • High-performance data access & transport • Grid Resource Information Service (GRIS) + Monitoring and Discovery Service (MDS) • Access to structure & state information • XIO • TCP, UDP, IP multicast, and file I/O BNCOD: Current Grids – July 6th, 2004 - 11 Convergence and co-operation 600 Condor jobs personal yourPool Condor workstation Condor Friendly Condor Pool PBS LSF Condor Globus Grid BNCOD: Current Grids – July 6th, 2004 - 12 6 Portals: Browser interfaces to Grid systems https+java/xml+rfb WEB Browser GENIUS Local WS EnginFrame Apache EDG UI EDG+GSI the Grid Roberto Barbera BNCOD: Current Grids – July 6th, 2004 - 13 UK e-Science Grid Globus Alliance e-Science Institute Guaranteed resources HPC(x) Digital Curation Centre Grid Operations & Support Centre CeSC (Cambridge) Open Middleware Infrastructure Institute www.nesc.ac.uk BNCOD: Current Grids – July 6th, 2004 - 14 7 Unfinished business • • • • • • • • • Provisioning Deployment, configuration and update Resource usage and billing Workflow description and enactment Resource description and discovery Workflow, reservation and advanced scheduling Quality of Service specification and assurance Security, trust and provenance … BNCOD: Current Grids – July 6th, 2004 - 15 Questions? BNCOD: Current Grids – July 6th, 2004 - 16 8