Grid Computing and the Open Grid Service Architecture Ian Foster Argonne National Laboratory University of Chicago http://www.mcs.anl.gov/~foster 2nd IEEE Intl Symp. on Network Computing & Applications, Boston, April 17, 2003 Partial Acknowledgements 3 Open Grid Services Architecture design – Carl Kesselman, Karl Czajkowski @ USC/ISI – Steve Tuecke @ANL – Jeff Nick, Steve Graham, Jeff Frey @ IBM Grid services collaborators at ANL – Kate Keahey, Gregor von Laszewski – Thomas Sandholm, Jarek Gawor, John Bresnahan Globus Toolkit R&D also involves many fine scientists & engineers at ANL, USC/ISI, and elsewhere (see www.globus.org) Strong links with many EU, UK, US Grid projects Support from DOE, NASA, NSF, IBM, Microsoft foster@mcs.anl.gov ARGONNE CHICAGO 4 Overview Grid: why and what Evolution of Grid technology – Open Grid Services Architecture Future directions – Towards lightweight VOs: dynamic trust relationships – Towards global knowledge communities: virtual data and dynamic workspaces foster@mcs.anl.gov ARGONNE CHICAGO Why the Grid? (1) Revolution in Science 5 Pre-Internet – Theorize &/or experiment, alone or in small teams; publish paper Post-Internet – Construct and mine large databases of observational or simulation data – Develop simulations & analyses – Access specialized devices remotely – Exchange information within distributed multidisciplinary teams foster@mcs.anl.gov ARGONNE CHICAGO Why the Grid? (2) Revolution in Business 6 Pre-Internet – Central data processing facility Post-Internet – Enterprise computing is highly distributed, heterogeneous, inter-enterprise (B2B) – Business processes increasingly computing- & data-rich – Outsourcing becomes feasible => service providers of various sorts foster@mcs.anl.gov ARGONNE CHICAGO New Opportunities Demand New Technology 7 “Resource sharing & coordinated problem solving in dynamic, multiinstitutional virtual organizations” “When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances” foster@mcs.anl.gov (George Gilder) ARGONNE CHICAGO 8 Grid Communities & Technologies Yesterday – Small, static communities, primarily in science – Focus on sharing of computing resources – Globus Toolkit as technology base Today – Larger communities in science; early industry – Focused on sharing of data and computing – Open Grid Services Architecture emerging Tomorrow – Large, dynamic, diverse communities that share a wide variety of services, resources, data – New issues: Trust, distributed RM, knowledge foster@mcs.anl.gov ARGONNE CHICAGO 9 NSF TeraGrid NCSA, SDSC, Argonne, Caltech Unprecedented capability – 13.6 trillion flop/s – 600 terabytes of data – 40 gigabits per second – Accessible to thousands of scientists working on advanced research www.teragrid.org foster@mcs.anl.gov ARGONNE CHICAGO 10 foster@mcs.anl.gov ARGONNE CHICAGO Data Grids for High Energy Physics 11 Enable international community of 1000s to access & analyze petabytes of data Harness computing & storage worldwide Virtual data concepts: manage programs, data, workflow Distributed system management foster@mcs.anl.gov ARGONNE CHICAGO NEESgrid Earthquake Engineering Collaboratory 12 U.Nevada Reno www.neesgrid.org foster@mcs.anl.gov ARGONNE CHICAGO Grid Computing 13 Grid Computing By M. Mitchell Waldrop May 2002 Hook enough computers together and what do you get? A new kind of utility that offers supercomputer processing on tap. Is Internet history about to repeat itself? foster@mcs.anl.gov ARGONNE CHICAGO Industrial Perspective on Grids: A Wide Range of Applications 14 Grid Services Market Opportunity 2005 Unique by Industry with Common Characteristics Manufacturing Financial Services Energy Derivatives Analysis Seismic Analysis Statistical Analysis Reservoir Analysis Portfolio Risk Analysis Mechanical/ Electronic Design LS / Bioinformatics Other Process Simulation Cancer Research Entertainment Finite Element Analysis Drug Discovery Digital Rendering Protein Folding Massive Multi-Player Games Failure Analysis Protein Sequencing Streaming Media Web Applications Weather Analysis Code Breaking/ Simulation Academic “Gridified” Infrastructure Sources: IDC, 2000 and Bear Stearns- Internet 3.0 - 5/01 Analysis by SAI foster@mcs.anl.gov ARGONNE CHICAGO 15 Overview Grid: why and what Evolution of Grid technology – Open Grid Services Architecture Future directions – Towards lightweight VOs: dynamic trust relationships – Towards global knowledge communities: virtual data and dynamic workspaces foster@mcs.anl.gov ARGONNE CHICAGO 16 Open Grid Services Architecture Service-oriented architecture – Key to virtualization, discovery, composition, local-remote transparency Leverage industry standards – Internet, Web services Distributed service management – A “component model for Web services” A framework for the definition of composable, interoperable services “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman,ARGONNE Nick, Tuecke, 2002 foster@mcs.anl.gov CHICAGO 17 Web Services XML-based distributed computing technology Web service = a server process that exposes typed ports to the network Described by the Web Services Description Language, an XML document that contains – Type of message(s) the service understands & types of responses & exceptions it returns – “Methods” bound together as “port types” – Port types bound to protocols as “ports” A WSDL document completely defines a service and how to access it foster@mcs.anl.gov ARGONNE CHICAGO 18 OGSA Structure A standard substrate: the Grid service – Standard interfaces and behaviors that address key distributed system issues – A refactoring and extension of the Globus Toolkit protocol suite … supports standard service specifications – Resource management, databases, workflow, security, diagnostics, etc., etc. – Target of current & planned GGF efforts … and arbitrary application-specific services based on these & other definitions foster@mcs.anl.gov ARGONNE CHICAGO 19 Open Grid Services Infrastructure Client Introspection: • What port types? • What policy? • What state? Grid Service Handle handle resolution Grid Service Reference Lifetime management • Explicit destruction • Soft-state lifetime GridService (required) Data access Service data element Service data element Other standard interfaces: factory, notification, collections Service data element Implementation Hosting environment/runtime (“C”, J2EE, .NET, …) foster@mcs.anl.gov ARGONNE CHICAGO 20 Open Grid Services Infrastructure GWD-R (draft-ggf-ogsi- gridservice-23) Open Grid Services Infrastructure (OGSI) http://www.ggf.org/ogsi-wg Editors: S. Tuecke, ANL K. Czajkowski, USC/ISI I. Foster, ANL J. Frey, IBM S. Graham, IBM C. Kesselman, USC/ISI D. Snelling, Fujitsu Labs P. Vanderbilt, NASA February 17, 2003 Open Grid Services Infrastructure (OGSI) foster@mcs.anl.gov ARGONNE CHICAGO Example: Reliable File Transfer Service Client Client 21 Client Request and manage file transfer operations File Notf’n Policy Grid Service Transfer Source Fault Monitor Perf. Monitor Query &/or subscribe to service data Pending Performance Policy Faults interfaces service data elements Internal State Data transfer operations foster@mcs.anl.gov ARGONNE CHICAGO Open Grid Service Architecture: Next Steps 22 Technical specifications – Open Grid Services Infrastructure is complete – Security, data access, Java binding, common resource models, etc., etc., in the pipeline Implementations and compliant products – Here: OGSA-based Globus Toolkit v3, … – Announced: IBM, Avaki, Platform, Sun, NEC, HP, Oracle, UD, Entropia, Insors, …, … Rich set of service defns & implementations foster@mcs.anl.gov ARGONNE CHICAGO Globus Toolkit v3 (GT3) Open Source OGSA Technology Implements OGSI interfaces Supports primary GT2 interfaces 23 – High degree of backward compatibility Multiple platforms & hosting environments – J2EE, Java, C, .NET, Python New services – SLA negotiation, service registry, community authorization, data management, … Rapidly growing adoption and contributions: “Linux for the Grid” foster@mcs.anl.gov ARGONNE CHICAGO 24 Overview Grid: why and what Evolution of Grid technology – Open Grid Services Architecture Future directions – Towards lightweight VOs: dynamic trust relationships – Towards global knowledge communities: virtual data and dynamic workspaces foster@mcs.anl.gov ARGONNE CHICAGO 25 Future Directions Grids are about computers, certainly – “On-demand” access to computing, etc. – Challenging future issues here: e.g., scale foster@mcs.anl.gov ARGONNE CHICAGO 26 CMS Event Simulation Production Production Run on the Integration Testbed – Simulate 1.5 million full CMS events for physics studies: ~500 sec per event on 850 MHz processor – 2 months continuous running across 5 testbed sites – Managed by a single person at the US-CMS Tier 1 foster@mcs.anl.gov ARGONNE CHICAGO 27 CMS Event Simulation Production Production Run on the Integration Testbed – Simulate 1.5 million full CMS events for physics studies: ~500 sec per event on 850 MHz processor – 2 months continuous running across 5 testbed sites – Managed by a single person at the US-CMS Tier 1 foster@mcs.anl.gov ARGONNE CHICAGO 28 Future Directions Grids are about computers, certainly – “On-demand” access to computing, etc. – Challenging future issues here: e.g., scale But they are ultimately about people, their activities, and their interactions – New interaction modalities supported by ondemand formation of lightweight VOs – New technologies needed: e.g., trust, security, data and knowledge integration Convergence of interest between “Compute” and “Collaboration” Grids? foster@mcs.anl.gov ARGONNE CHICAGO Global Knowledge Communities foster@mcs.anl.gov 29 ARGONNE CHICAGO 30 Example Issue: Trust and Security Effective VO operation depends critically on – Trust: can I rely on you? – Protection mechanisms to govern actions Suffers from VO-organization policy mismatch Goal: collaborations no longer defined by slow centralized mechanisms but can – form spontaneously; – be managed in a distributed manner; and – be protected by an infrastructure that maintains and enforces trust relationships foster@mcs.anl.gov ARGONNE CHICAGO 31 Grid Security Services Requestor's Domain Trust Service Attribute Service Audit/ Secure-Logging Service Service Provider's Domain Authorization Service Authorization Service Privacy Service Trust Service Attribute Service Audit/ Secure-Logging Service Privacy Service Credential Validation Service Credential Validation Service Bridge/ Translation Service Requestor Application WS-Stub Secure Conversation WS-Stub Credential Validation Service Service Provider Application Credential Validation Service Authorization Service Authorization Service Attribute Service Attribute Service Trust Service Trust Service VO Domain foster@mcs.anl.gov ARGONNE CHICAGO Understanding and Enhancing VO Trust and Security Usability analysis Community Social network analysis Other analyses trust ( Tr, Te, As, L ) <- Cs; recommend ( Rr, Re, As, L ) <- Cs; Trust Monitoring for reputation, compliance, intrusion detection, etc. Establishment, enhancement, maintenance, verification Workflow analysis Risk analysis allowed (S, O, A, C) Policy Factoring wrt environment Mechanism foster@mcs.anl.gov 32 Feasibility analysis wrt cost, legality, etc. VOTA, PKI, VPN, etc. ARGONNE CHICAGO Virtual Data for Collaborative Science 33 Much collaboration is concerned with the development & use of knowledge, whether – Programs for data analysis and generation – Computations involving those programs – Metadata concerning data, programs, computations—and their interrelationships In a distributed, heterogeneous, fractal (?) environment with widely varying – Data and analysis program formats – Degrees of formality and scale – Scientific goals and sharing policies foster@mcs.anl.gov ARGONNE CHICAGO Sloan Digital Sky Survey Production System foster@mcs.anl.gov 34 ARGONNE CHICAGO Virtual Data Concept 35 Capture and manage information about relationships among – Data (of widely varying representations) – Programs (& their execution needs) – Computations (& execution environments) Apply this information to, e.g. – Discovery: Data and program discovery – Workflow: Structured paradigm for organizing, locating, specifying, & requesting data – Explanation: provenance – Planning and scheduling – Other uses we haven’t thought of foster@mcs.anl.gov ARGONNE CHICAGO 36 “I’ve come across some interesting data, but I need to understand the nature of the corrections applied when it was constructed before I can trust it for my purposes.” Motivations Data created-by Transformation execution-of “I want to search an astronomical database for galaxies with certain characteristics. If a program that performs this analysis exists, I won’t have to write one from scratch.” foster@mcs.anl.gov “I’ve detected a calibration error in an instrument and want to know which derived data to recompute.” consumed-by/ generated-by Derivation “I want to apply an astronomical analysis program to millions of objects. If the results already exist, I’ll save weeks of computation.” ARGONNE CHICAGO Example: Sloan Galaxy Cluster Analysis 37 DAG Sloan Data 100000 Galaxy cluster size distribution Number of Clusters 10000 1000 100 10 1 foster@mcs.anl.gov 1 10 Number of Galaxies 100 Jim Annis, Steve Kent, Vijay Sehkri, Fermilab; Michael ARGONNE CHICAGO Milligan, Yong Zhao, Chicago 38 Integrating Provenance Data Group Index Personal VDS DV Personal Index Personal Index DS TR Collaborationlevel index DV DV TR DS TR TR Collaboration VDS DV Group VDS DS Personal Index DV DV Personal VDS Collaboration-wide index foster@mcs.anl.gov ARGONNE CHICAGO 39 Summary Yesterday – Small, static communities, primarily in science – Focus on sharing of computing resources – Globus Toolkit as technology base Today – Larger communities in science; early industry – Focused on sharing of data and computing – Open Grid Services Architecture emerging Tomorrow – Large, dynamic, diverse communities that share a wide variety of services, resources, data – New issues: Trust, distributed RM, knowledge foster@mcs.anl.gov ARGONNE CHICAGO 40 For More Information The Globus Project™ – www.globus.org Technical articles – www.mcs.anl.gov/~foster Open Grid Services Arch. – www.globus.org/ogsa Chimera – www.griphyn.org/chimera Global Grid Forum – www.gridforum.org foster@mcs.anl.gov ARGONNE CHICAGO