ICENI Overview & Grid Scheduling Laurie Young London e-Science Centre Department of Computing, Imperial College ICENI The Iceni, under Queen Boudicca, united the tribes of South-East England in a revolt against the occupying Roman forces in AD60. IC e-Science Networked Infrastructure Developed by LeSC Grid Middleware Group Collect and provide relevant Grid meta-data Use to define and develop higher-level services Interaction with other frameworks: OGSA, Jxta etc. 2 ICENI (The Big Picture) Web Services Gateway Public Computational Community Computational Resource JavaCoG Globus Storage Resources Network Resources CR SR Identity Manager Private Administrative Domain CR Applicatio n Portal Resource Browser Domain Manager CR SR SR Public Computational Community SR Software Resources Resource Broker Application Mapper Policy Manager CR Resource Manager Private SR Gateway between private and public regions Component Design Tools Application Design Tools Public 3 ICENI Stack Portal Interface Application Construction & Deployment ICENI Middleware Grid Fabric 4 Web Portals • Handheld wireless devices become ubiquitous – Personal Digital Assistants, Mobile Phones – Secure access any time, any place, any where • Use X.509 certificates embedded in a browser to authenticate user’s identity • Integration portal infrastructure with ICENI – EPIC: Use component meta-data to build portal application • Goal: Provide secure ‘one stop shop’ for escience 5 EPIC: e-Science Portal at Imperial College • Collaborative LeSC industrial project with Sun Microsystems • Develop a secure portal infrastructure to: – Access your own personal environment – Applications to support day-to-day e-science – Interaction with other Grid infrastructures • Allow role based access to resources – Anonymous: public web pages – Students: internal pages, email, compute resources – Staff: restricted pages 6 ICENI Application Model • Legacy code! • Component Applications – Compose applications from many components – Component does work on data – Component communicates data 7 Component Motivation • • • • • Logical application model Collaborative software authoring Promote component reuse and sharing Simplify application construction Enable deployment to diverse Grid resources: – Communication Selection – Implementation Selection 8 Layered Abstraction Meaning dataflow abstract data types may have many Behaviour Behaviour control flow threads etc. may each have many Implementation Implementation performance, architectures, concrete data type 9 Component’s View of the Grid Context object Other Code SOAP RMI You may call methods provided by the middleware More Code My Code You must implement a provided interface 10 Visual Component Composition 11 Deployment of Components End Application Design User Tools Repository Code Code Access Resource Information Application Description Document Code Component Implementation Design Annotating Tools Tools Application Mapper Run-Time Representation APO Code Application Proxy Object Code RTR Grid 12 Component Execution OGSA, Jxta, etc. OGSA, Jxta, etc. APO Jini Jini MPI Code Code Code RTR RTR RTR SOAP RMI Compute Resource Network Resource Hardware 13 Components as Services Service interface Context object SOAP (or other) protocol Component 14 ICENI & Jini: P2P Conceptual Model of peer- to- peer architecture Service requester Two- way Interaction with service Lookup Service Service Matches Register service Service Locator Service Provider Discover services 15 Web Services Architecture Web Service Model Service requester (SOAP Client) SOAP message exchange Service Provider (SOAP Server) UDDI Lookup UDDI Response, WSDL location UDDI registration Web Services Crawler? Service Locator (UDDI Registry) 16 Synergy Service requester The Synergy Service Requester Web Service Client SOAP Proxy SOAP Java App SOAP P2P (Jini) Jini Object RMI Jini Lookup RMI SOAP Service Provider Web Service Proxy Web Services Service Locator Web Service Proxy 17 Grid Service Contracts Jini Lookup Service DRMAA Client DRMAA Resource 18 Grid Service Contracts Resource Browser DRMAA Client User:B Jini Lookup Service DRMAA Resource User: A+B Duration:1hr19 Grid Service Contracts User:A Duration:10m DRMAA Resource User:A DRMAA Client Jini Lookup Service DRMAA Client User:B DRMAA Resource User: A+B Duration:1hr20 OGSA & Jini Integration User:A Duration:10m DRMAA Resource Gateway Manager GSI enabled Web Service Hosting Environment Jini Lookup Service DRMAA Resource User: A+B Duration:1hr21 OGSA & Jini Integration User:A Duration:10m DRMAA Resource Gateway Manager WSDL Interface GSI enabled Web Service Hosting Environment Jini Lookup Service Jini Client Interface DRMAA Resource User: A+B Duration:1hr22 OGSA & Jini Integration User:A Duration:10m DRMAA Resource Gateway Manager WSDL Interface GSI + SOAP Connection GSI enabled Web Service Hosting Environment Jini Lookup Service Jini Client Interface DRMAA Resource User: A+B Duration:1hr23 OGSA & Jini Integration User:A Duration:10m DRMAA Resource Gateway Manager WSDL Interface GSI enabled Web Service Hosting Environment Jini Lookup Service SOAP->Java GSI + SOAP Connection User Info Jini Client Interface DRMAA Resource User: A+B Duration:1hr24 Application Mapping (Scheduling) • Architecture – How meta-data is collected – What meta-data is required • Scheduling Algorithms – Map components onto resources for “best” results – Meta-data dependent decisions 25 Scheduling Architecture ICENI App Builder (GUI) Component Repository Performance Models Scheduler Launcher Resources 26 Multiple Metrics (1) • “It is the goal of a scheduler to optimise one or more metrics” (Feitelson & Rudolph) • Generally one metric is most important – Application Optimisation • Execution time • Execution cost – Host Optimisation • Host utilisation • Host throughput • Interaction Latency 27 Multiple Metrics (2) • In a Grid Environment there are three application optimisation based important metrics – Start time (b ) – End time (e ) – Cost ( ) • Relative importance varies on a user by user and application by application basis 28 Combining Metrics – Benefit Fn • A Benefit Function maps the metrics we are interested in to a single Benefit Value metric • Different benefit functions represent different optimisation preferences B B(b,e, ) 29 Optimisation Preferences • Cost Optimisation • Time Optimisation B if e e max and max B if e e max and max • Cost/Time Optimisation B if e e max and max 30 Schedule Benefit • Each component and communication has a benefit function • Each resource and network connection has a predicted time & cost for each component or communication that could be deployed • Fit the tasks onto the resources to get the maximum Total Predicted Benefit Bt B(b,e, ) 31 Graph Oriented Scheduling (1) • Applications are described as a graph – Nodes represent application components – Edges represent component communication • Resources are described as a graph – Nodes represent resources – Edges represent network connections 32 Graph Oriented Scheduling (2) Design Factory Analyse Atlas Saturn Scatter Viking Mesh Mesh Mesh DRACS DRACS DRACS Condor pool Gather 33 Graph Oriented Scheduling (3) Atlas Design Condor pool Gather Scatter Saturn Analyse Viking Factory 34 Summary • Component framework provides: – Rich application meta-data – Decoupled component definition and implementation • Meta-Data: – Exploit performance information to map component implementation to the ‘best’ resources • Resource Broker: – Resource selection through user defined policies: • Minimise cost using computational economics • Minimise execution time using the application mapper 35 Acknowledgements • Director: Professor John Darlington • Technical Director: Dr Steven Newhouse • Research Staff: – – – – – Anthony Mayer, Nathalie Furmento Stephen McGough, James Stanton Yong Xie, William Lee Marko Krznaric, Murtaza Gulamali Asif Saleem, Laurie Young, Gary Kong • Contact: – http://www.lesc.ic.ac.uk/ – e-mail: lesc@ic.ac.uk 36