Information-Driven Science in Pervasive Grid Environments Manish Parashar The Applied Software Systems Laboratory ECE/CAIP, Rutgers University http://www.caip.rutgers.edu/TASSL (Ack: NSF, DoE, NIH) Outline • Pervasive Grid Environments - Unprecedented Opportunities • Pervasive Grid Environments - Unprecedented Challenges, Opportunities • Project AutoMate @ TASSL, Rutgers University – Enabling Autonomic Applications in Pervasive Grid Environments • An Illustrative Application • Concluding Remarks Pervasive Grid Environments - Unprecedented Opportunities • Pervasive Grids Environments – Seamless, secure, on-demand access to and aggregation of, geographically distributed computing, communication and information resources • Computers, networks, data archives, instruments, observatories, experiments, sensors/actuators, ambient information, etc. – Context, content, capability, capacity awareness – Ubiquity and mobility • Knowledge-based, information/data-driven, context/content-aware computationally intensive science – Symbiotically and opportunistically combine computations, experiments, observations, and real-time information to model, manage, control, adapt, optimize, … • Crisis management, monitor and predict natural phenomenon, monitor and manage engineering systems, optimize business processes • A new paradigm for scientific investigation ? – seamless access • resources, services, data, information, expertise, … – seamless aggregation – seamless (opportunistic) interactions/couplings The original Grid concept has moved on! • Coordinated resource sharing and problem solving in dynamic, multiinstitutional virtual organizations. Source: I. Foster et al Pervasive Grid Environments and Information Driven Science Resources discovered, negotiated, co-allocated on-the-fly. Model/Simulation deployed Experts query, configure resources Experts interact and collaborate using ubiquitous and pervasive portals Experts monitor/interact with/interrogate/steer models (“what if” scenarios,…). Application notifies experts of interesting phenomenon. Models write into the archive Real-time data assimilation/injection (sensors, instruments, experiments, data archives), Experts mine archive, match real-time data with history Automated mining & matching Models dynamically composed. “WebServices” discovered & invoked. Information-driven Management of Subsurface Geosystems: The Instrumented Oil Field (with UT-CSM, UT-IG, OSU, UMD, ANL) Model Driven Detect and track changes in data during production. Invert data for reservoir properties. Detect and track reservoir changes. Assimilate data & reservoir properties into the evolving reservoir model. Use simulation and optimization to guide future production. Data Driven Dynamic, Data Driven Reservoir Management Dynamic Decision System Dynamic Data-Driven Assimilation Optimize • Economic revenue Management decision • Environmental hazard •… Subsurface characterization Based on the present subsurface knowledge and numerical model Improve knowledge of subsurface to reduce uncertainty Data assimilation Update knowledge of model Acquire remote sensing data Improve numerical model Plan optimal data acquisition Experimental design START Vision: Diverse Geosystems – Similar Solutions Landfills Oilfields Models Simulation Underground Pollution Control Data Undersea Reservoirs Management of the Ruby Gulch Waste Repository (with UT-CSM, INL, OU) • Ruby Gulch Waste Repository/Gilt Edge Mine, South Dakota – ~ 20 million cubic yard of waste rock – AMD (acid mine drainage) impacting drinking water supplies • Monitoring System – Multi electrode resistivity system (523) • One data point every 2.4 seconds from any 4 electrodes – Temperature & Moisture sensors in four wells “Towards Dynamic Data-Driven Management of the Ruby Gulch Waste Repository,” M. Parashar, et al, DDDAS Workshop, ICCS 2006, Reading, UK, LNCS, Springer Verlag, Vol. 3993, pp. 384 – 392, May 2006. – Flowmeter at bottom of dump – Weather-station – Manually sampled chemical/air ports in wells – Approx 40K measurements/day Dynamic Data-Driven Waste Management Sensors Optimization Algorithms Ruby Gulch Waste Repository Actuators Controllable input Data Assimilation Optimization Observations Control algorithms Models, Surrogate/ Reduced models Adaptive Fusion of Stochastic Information for Imaging Fractured Vadose Zones (with U of AZ, OSU, U of IW) • Near-Real Time Monitoring, Characterization and Prediction of Flow Through Fractured Rocks System responses Inverse Modeling Parameters, Boundary & Initial Conditions Forward Modeling Prediction bad Network design Comparison With observations good Application Data-Driven Forest Fire Simulation (U of AZ) • Predict the behavior and spread of wildfires (intensity, propagation speed and direction, modes of spread) – based on both dynamic and static environmental and vegetation conditions – factors include fuel characteristics and configurations, chemical reactions, balances between different modes of hear transfer, topography, and fire/atmosphere interactions. “Self-Optimizing of Large Scale Wild Fire Simulations,” J. Yang*, H. Chen*, S. Hariri and M. Parashar, Proceedings of the 5th International Conference on Computational Science (ICCS 2005), Atlanta, GA, USA, Springer-Verlag, May 2005. System for Laser Treatment of Cancer – UT, Austin Source: L. Demkowicz, UT Austin Synthetic Environment for Continuous Experimentation – Purdue University Source: A. Chaturvedi, Purdue Univ. Integrated Wireless Phone Based Emergency Response System – Notre Dame • Detect abnormal patterns in mobile call activity and locations • Initiate dynamic data driven simulations to predict the evolution of the abnormality • Initiate higher resolution data collection in localities of interest • Interface with emergency response Decision Support Systems Source: G. Madey, ND Many Application Areas …. • Hazard prevention, mitigation and response – Earthquakes, hurricanes, tornados, wild fires, floods, landslides, tsunamis, terrorist attacks • Critical infrastructure systems – Condition monitoring and prediction of future capability • Transportation of humans and goods – Safe, speedy, and cost effective transportation networks and vehicles (air, ground, space) • Energy and environment – Safe and efficient power grids, safe and efficient operation of regional collections of buildings • Health – Reliable and cost effective health care systems with improved outcomes • Enterprise-wide decision making – Coordination of dynamic distributed decisions for supply chains under uncertainty • Next generation communication systems – Reliable wireless networks for homes and businesses • ………… • Report of the Workshop on Dynamic Data Driven Applications Systems, F. Darema et al., March 2006, www.dddas.org Source: M. Rotea, NSF Outline • Pervasive Grid Environments - Unprecedented Opportunities • Pervasive Grid Environments - Unprecedented Challenges – Uncertainty • System uncertainty • Application uncertainty • Information uncertainty • Project AutoMate @ TASSL, Rutgers University – Enabling Autonomic Applications in Pervasive Grid Environments • An Illustrative Application • Concluding Remarks Pervasive Grid Applications – Unprecedented Challenges: Uncertainty • System Uncertainty – Very large scales – Ad hoc (amorphous) structures/behaviors • p2p, hierarchical, etc, architectures – Dynamic • entities join, leave, move, change behavior – Heterogeneous • capability, connectivity, reliability, guarantees, QoS – Lack of guarantees • components, communication – Lack of common/complete knowledge • number, type, location, availability, connectivity, protocols, semantics, etc. Pervasive Grid Applications – Unprecedented Challenges: Uncertainty • Application Uncertainty – Dynamic behaviors • space-time adaptivity – Dynamic and complex couplings • multi-physics, multi-model, multi-resolution, …. – Dynamic and complex (ad hoc, opportunistic) interactions • application application, application resource, application data, application user, … – Software/systems engineering issues • Emergent rather than by design • Information Uncertainty – Availability, resolution, quality of information – Devices capability, operation, calibration – Trust in data, data models Pervasive Grid Applications – Research Issues, Opportunities • Applications and algorithms – tobust model/algorithm development and calibration • impact of information on models and models on information acquisition – – – – continuous model, algorithm, emergent system validation uncertainty estimation parameter selection and optimization observability, identifiability, tractability • Measurement and actuation systems – “real-time” data collection and transport • in-network aggregation, assimilation – – – – data selection and application integration, data quality management data/data-model heterogeneity security trust, data provenance, audit trails actuation and control Pervasive Grid Applications – Research Issues, Opportunities • Systems software – programming systems/models for data integration and runtime selfmanagement • components and compositions capable of adapting behavior and interactions • correctness, consistency, performance, quality-of-service constraints – data management mechanisms for acquisition with real time, space and data quality constraints • high data volumes/rates, heterogeneous data qualities, sources – runtime execution services that guarantee correct, reliable execution with predictable and controllable response time • data assimilation, injection, adaptation Outline • Pervasive Grid Environments - Unprecedented Opportunities • Pervasive Grid Environments - Unprecedented Challenges, Opportunities • Project AutoMate @ TASSL, Rutgers University – Enabling Autonomic Applications in Pervasive Grid Environments • An Illustrative Application • Concluding Remarks Programming Pervasive Grid Systems – Programming System • programming model, languages/abstraction – syntax + semantics – entities, operations, rules of composition, models of coordination/communication • abstract machine, execution context and assumptions • infrastructure, middleware and runtime Applications Programming System (Programming Model + Abstract Machine) Abstraction (Support abstract behaviors/assumptions) Grid Middleware Infrastructure Virtualization (Create and manage VOs and VMs) Virtual Machine Virtual Machine Virtual Machine – Hide or expose uncertainty? • robustness, ease of programming • the inverted stack … Virtual Organization Organization Organization Organization “Conceptual and Implementation Models for the Grid,” M. Parashar and J.C. Browne, Proceedings of the IEEE, Special Issue on Grid Computing, IEEE Press, Vol. 19, No. 3, March 2005. Programming Pervasive Grid Systems • Computing has evolved and matured to provide specialized solutions to satisfy relatively narrow and well defined requirements in isolation – performance, security, dependability, reliability, availability, throughput, pervasive/amorphous, automation, reasoning, etc. • In case of pervasive Grid applications/environments, requirements, objectives, execution contexts are dynamic and not known a priori – requirements, objectives and choice of specific solutions (algorithms, behaviors, interactions, etc.) depend on runtime state, context, and content – applications should be aware of changing requirements and executions contexts and to respond to these changes are runtime • Autonomic computing - systems/applications that self-manage – use appropriate solutions based on current state/context/content and based on specified policies – address uncertainty at multiple levels – asynchronous algorithms, decoupled interactions/coordination, content-based substrates Ontology, Taxonomy Programming System Autonomic Components, Dynamic Composition, Opportunistic Interactions, Collaborative Monitoring/ Control Decentralized Coordination Engine Agent Framework, Decentralized Reactive Tuple Space Semantic Middleware Services Content-based Discovery, Associative Messaging Content Overlay Content-based Routing Engine, Self-Organizing Overlay Rudder Coordination Middleware Meteor/Squid Content-based Middleware Autonomic Grid Applications Sesame/DAIS Protection Service Accord Programming Framework Project AutoMate: Enabling Autonomic Applications • Conceptual models and implementation architectures – programming systems based on popular programming models • object, component and service based prototypes – content-based coordination and messaging middleware – amorphous and emergent overlays • http://automate.rutgers.edu Project AutoMate: Components • Accord – A Programming System for Autonomic Grid Applications • Rudder/Comet – Decentralized Coordination Middleware • Meteor – Content-based Interactions Middleware • ACE – Autonomic Composition Engine • Squid – Decentralized Information Discovery and Contentbased Routing • SESAME – Context-Aware Access Management • DAIS – Cooperative Protection against Network Attacks • More information/Papers – http://automate.rutgers.edu “AutoMate: Enabling Autonomic Grid Applications,” M. Parashar et al, Cluster Computing: The Journal of Networks, Software Tools, and Applications, Special Issue on Autonomic Computing, Kluwer Academic Publishers. Vol. 9, No. 2, pp. 161 – 174, 2006. Accord: Rule-Based Programming System • Accord is a programming system which supports the development of autonomic applications. – Enables definition of autonomic components with programmable behaviors and interactions. – Enables runtime composition and autonomic management of these components using dynamically defined rules. • Dynamic specification of adaptation behaviors using rules. • Enforcement of adaptation behaviors by invoking sensors and actuators. • Runtime conflict detection and resolution. • 3 Prototypes: Object-based, Components-based (CCA), Service-based (Web Service) “Accord: A Programming Framework for Autonomic Applications,” H. Liu* and M. Parashar, IEEE Transactions on Systems, Man and Cybernetics, Special Issue on Engineering Autonomic Systems, IEEE Press, Vol. 36, No 3, pp. 341 – 352, 2006. Autonomic Element in Accord Event generation Element Manager Other Interface invocation Actuator invocation Functional Port Computational Element Control Port Element Manager Operational Port Autonomic Element Internal state Rules Contextual state The Accord Runtime Infrastructure Application strategies Application requirements Application workflow Composition manager Interactio n rules Interactio n rules Interactio n rules Interactio n rules Behavior rules Behavior rules Behavior rules Behavior rules LLC Based Self Management within Accord LLC Controller Self-Managing Element LLC Controller Optimization Function Computational Element Model Advice Element Manager Element Manager Computational Element Contextual State Internal State • Element/Service Managers are augmented with LLC Controllers – monitors state/execution context of elements – enforces adaptation actions determined by the controller – augment human defined rules Decentralized (Decoupled/Asynchronous) Content-based Middleware Services SquidTON: Reliability and Fault Tolerance • Pervasive Grid systems are dynamic, with nodes joining, leaving and failing relatively often • • => data loss and temporarily inconsistent overlay structure => the system cannot offer guarantees – Build redundancy into the overlay network – Replicate the data • SquidTON = Squid Two-tier Overlay Network – Consecutive nodes form unstructured groups, and at the same time are connected by a global structured overlay (e.g. Chord) Data is replicated in the group – 10 17 22 10 17 25 249 240 231 33 249 22 240 33 41 82 192 86 185 91 94 192 25 231 250 82 185 86 102 128 161 94 91 132 157 141 157 150 150 141 132 128 Groups of nodes Group identifier Content Descriptors and Information Space • Data element = a piece of information that is indexed and discovered – Data, documents, resources, services, metadata, messages, events, etc. network computer 2D keyword space for a P2P file sharing system Base bandwidth (Mbps) t os C 9 30 Storage space (MB) Complex query (comp*, *) comp* keyword1 Computational resource 100 keyword2 Document keyword1 Complex query (10, 20-25, *) 20 Base bandwidth (Mbps) keyword2 • Each data element has a set of keywords associated with it, which describe its content => data elements form a keyword space 25 3D keyword space for resource sharing, using the attributes: storage space, base bandwidth and cost Co 10 st Storage space (MB) Content Indexing: Hilbert SFC • f: Nd N, recursive generation 01 10 1 0101 0100 0110 1000 0111 0011 0010 00 11 0 0000 0001 0 1 00 01 1011 1100 1110 10 1001 11 1010 10 1101 01 1111 00 11 • Properties: – Digital causality – Locality preserving – Clustering Cluster: group of cells connected by a segment of the curve Content Routing/Discovery - Squid Bandwidth Query: (Storage space = 30, Bandwidth = *) => Binary query (011110, *) 1 11 0101 0110 1001 0111 01 1000 101 1011 00 0 1 Storage space … 100 011 0011 0010 1101 0000 0001 1110 1111 1100 010 11 00 (011, *) 110 10 01 10 0100 0 111 1010 001 00 01 10 000 000 001 010 011 100 101 110 111 11 00 0 000000 0 111000 0001 01 00, 01 000100 001 0001, 0010 100001 0110, 0111 001001 000101, 000110 011 0111 001111 001001, 001010 011111 011010, 011011, 011100 Query Engine – Experimental Evaluation • System size: 103 to 106 nodes • Data: – Uniformly distributed, synthetic generated data – 4*105 CiteSeer data • Load balanced system • Experiments: – Number of clusters generated for a query – Number of nodes queried • All results are plotted on a logarithmic scale Squid Content Routing/Discovery Engine: Optimization • Number of clusters generated for queries with coverage 1%, 0.1%, 0.01%, with and without optimization • The results are normalized against the clusters that the query defines on the curve (i.e. without optimization). 10 Normalized number of clusters Normalized number of clusters 10 1 0.1 0.01 0.001 0.0001 10 3 10 4 10 5 System size Clusters 1% query 0.1% query 0.01% query 3D Uniformly distributed data 1 0.1 0.01 0.001 0.0001 0.00001 10 3 Clusters 10 4 System size 1% query 0.1% query 3D CiteSeer data 10 5 0.01% query Squid Content Routing/Discovery Engine – Nodes Queried • Percentage of nodes queried for queries with coverage 1%, 0.1%, 0.01%, with and without optimization 10 % of nodes queried % of nodes queried 10 1 0.1 0.01 10 3 10 4 10 5 10 6 System size 1% 1% optimization 0.1% 0.1% optimization 1 0.1 0.01 10 3 10 4 1% 1% optimization 0.1% 0.1% optimization 10 5 System size 0.01% 0.01% optimization 3D Uniformly distributed data 3D CiteSeer data 0.01% 0.01% optimization Semantics of Associative Rendezvous Interactions profile credentials • Messages Header – (header, action, data) – Symmetric post primitive: does not differentiating between interest/data message context TTL (time to live) Action • Associative selection [Data] – match between interest and data profiles • store • retrieve • notify • delete Profile = list of (attribute, value) pairs: • Reactive behavior Example: <(sensor_type, temperature), (latitude, 10), (longitude, 20)> – Execute action field upon matching post (<p1, p2>, store, data) <p1, p2> (1) C1 <p1, *> match (2) post(<p1, *>, notify_data(C2) ) notify_data(C2) (3) notify(C2) C2 Comet Coordination Space • A virtual global shared-space is constructed from a semantic multi-dimensional information space, which is deterministically mapped onto the system peer nodes • The space is associatively accessible by all system peer nodes. Access is independent of the physical locations of tuples or hosts – Tuple distribution • A tuple/template is associated with k keywords • Squid content-based routing engine used or exact and approximate tuple distribution and retrieval – Transient spaces • Enable application to explicitly exploit context locality “COMET: A Scalable Coordination Space in Decentralized Distributed Environments,” Z. Li* and M. Parashar, Proceedings of the 2nd International Workshop on Hot Topics in Peer-to-Peer Systems (HOT-P2P 2005), San Diego, CA, USA, IEEE Computer Society Press, pp. 104 – 111, July 2005. Coordination primitives • Basic primitives – Out, In, Rd • Tuple retrieval – Exact retrieval • Keys only consist of complete keywords • Routs to a single destination – Approximate retrieval • Keys consist of partial keywords, wildcards • Routs to multiple destinations Supporting the Rudder Agent Framework • Agents communication – associatively reading, writing, and extracting tuples • Agent coordination protocols – Decentralized election protocol • Based on wait-free consensus protocols – Resilient to node/link failures – Discovery protocol • • • • Registry implemented using XML tuples Element registered using Out Element unregistered using In Elements discovered using Rd/RdAll operation – Interaction protocol • Contract-Net protocol • Two agent bargaining protocol • Workflow engine “Enabling Dynamic Composition and Coordination of Autonomic Applications using the Rudder Agent Framework,” Z. Li* and M. Parashar, The Knowledge Engineering Review, Cambridge University Press (also SAACS 2005). Implementation/Deployment Overview • Current implementation builds on JXTA – SquidTON, Squid, Comet and Meteor layers are implemented as event-driven JXTA services • Deployments include – Campus Grid @ Rutgers – Orbit wireless testbed (400 nodes) – PlanetLab wide-area testbed • At least one node selected from each continent Outline • Pervasive Grid Environments - Unprecedented Opportunities • Pervasive Grid Environments - Unprecedented Challenges, Opportunities • Project AutoMate @ TASSL, Rutgers University – Enabling Autonomic Applications in Pervasive Grid Environments • An Illustrative Application • Concluding Remarks The Instrumented Oil Field of the Future (UT-CSM, UT-IG, RU, OSU, UMD, ANL) • • Production of oil and gas can take advantage of installed sensors that will monitor the reservoir’s state as fluids are extracted Knowledge of the reservoir’s state during production can result in better engineering decisions – economical evaluation; physical characteristics (bypassed oil, high pressure zones); productions techniques for safe operating conditions in complex and difficult areas Detect and track changes in data during production Invert data for reservoir properties Detect and track reservoir changes Assimilate data & reservoir properties into the evolving reservoir model Use simulation and optimization to guide future production, future data acquisition strategy “Application of Grid-Enabled Technologies for Solving Optimization Problems in Data-Driven Reservoir Studies,” M. Parashar, H. Klie, U. Catalyurek, T. Kurc, V. Matossian, J. Saltz and M Wheeler, FGCS. The International Journal of Grid Computing: Theory, Methods and Applications (FGCS), Elsevier Science Publishers, Vol. 21, Issue 1, pp 19-26, 2005. Effective Oil Reservoir Management: Well Placement/Configuration • Why is it important – Better utilization/cost-effectiveness of existing reservoirs – Minimizing adverse effects to the environment Bad Management Better Management Much Bypassed Oil Less Bypassed Oil Effective Oil Reservoir Management: Well Placement/Configuration • What needs to be done – Exploration of possible well placements and configurations for optimized production strategies – Understanding field properties and interactions between and across subdomains – Tracking and understanding long term changes in field characteristics • Challenges – – – – Geologic uncertainty: Key engineering properties unattainable Large search space: Infinitely many production strategies possible Complex physical properties and interactions. Complex numerical models An Autonomic Well Placement/Configuration Workflow Generate Guesses Start Parallel Instance connects to DISCOVER IPARS Instances Send Guesses DISCOVER Notifies Clients Clients interact with IPARS SPSA Send guesses VFSA DISCOVER Optimization Service IPARS Factory client Exhaustive Search If guess in DB: send response to Clients and get new guess from Optimizer History/ Archived Data client MySQL Database If guess not in DB instantiate IPARS with guess as parameter Sensor/C ontext Data AutoMate Programming System/Grid Middleware Oil prices, weather,etc. Autonomic Oil Well Placement/Configuration Contours of NEval(y,z,500)(10) permeability Pressure contours 3 wells, 2D profile Requires NYxNZ (450) evaluations. Minimum appears here. VFSA solution: “walk”: found after 20 (81) evaluations Autonomic Oil Well Placement/Configuration (VFSA) “An Reservoir Framework for the Stochastic Optimization of Well Placement,” V. Matossian, M. Parashar, W. Bangerth, H. Klie, M.F. Wheeler, Cluster Computing: The Journal of Networks, Software Tools, and Applications, Kluwer Academic Publishers, Vol. 8, No. 4, pp 255 – 269, 2005 “Autonomic Oil Reservoir Optimization on the Grid,” V. Matossian, V. Bhat, M. Parashar, M. Peszynska, M. Sen, P. Stoffa and M. F. Wheeler, Concurrency and Computation: Practice and Experience, John Wiley and Sons, Volume 17, Issue 1, pp 1 – 26, 2005. Outline • Pervasive Grid Environments - Unprecedented Opportunities • Pervasive Grid Environments - Unprecedented Challenges, Opportunities • Project AutoMate @ TASSL, Rutgers University – Enabling Autonomic Applications in Pervasive Grid Environments • An Illustrative Application • Concluding Remarks Conclusion • Pervasive Grid Environments & Next Generation Scientific Investigation – Knowledge-based, data and information driven, context-aware, computationally intensive – Unprecedented opportunity for global scientific investigation • can enable accurate solutions to complex applications; provide dramatic insights into complex phenomena – Unprecedented research challenges • scale, complexity, heterogeneity, dynamism, reliability, uncertainty, … • applications, algorithms, measurements, data/information, software – Project AutoMate: Autonomic Computational Science on the Grid • semantic + autonomics • Accord, Rudder/Comet, Meteor, Squid, Topos, … • More Information, publications, software – www.caip.rutgers.edu/~parashar/ – parashar@caip.rutgers.edu The Team • TASSL, CAIP/ECE Rutgers University – – – – – – – – – Viraj Bhat Sumir Chandra Andres Q. Hernandez Nanyan Jiang Zhen Li (Jenny) Vincent Matossian Cristina Schmidt Mingliang Wang Li Zhang • Key Applications Collaborators – • – – Key CE/CS Collaborators – Rutgers Univ. • – Univ. of Arizona • – T. Kurc, J. Saltz GA Tech • – S. Hariri Ohio State Univ. • – D. Silver, D. Raychaudhuri, P. Meer, M. Bushnell, etc. K. Schwan, M. Wolf University of Maryland • A. Sussman, C. Hansen T. –C. J. Yeh, J. Daniels, A. Kruger Idaho National Laboratory • R. Versteeg PPPL • – J. Ray, J. Steensland Univ. of Arizona/Univ. of Iowa, OSU • – S. Klasky, C.S. Chang CRL, Sandia National Lab., Livermore • – H. Klie, M. Wheeler, M. Sen, P. Stoffa ORNL, NYU • – D. Foran, M. Reisse CSM/IG, Univ. of Texas at Austin • – R. Levy, S. Garofilini UMDNJ • – • Rutgers Univ. R. Samtaney ASCI/CACR, Caltech • J. Cummings Thank you !