SODA: Service-On-Demand Architecture for Application Service Hosting Utility Platforms Dongyan Xu, Xuxian Jiang Lab FRIENDS (For Research In Emerging Network and Distributed Services) Department of Computer Sciences Center for Education and Research in Information Assurance and Security (CERIAS) Purdue University Outline Motivations and goals Related work Research components of SODA Summary and on-going work Motivations Vision of utility computing Computation utility Storage utility Application service hosting Conference management e-Campaign Digital government Serving the underserved communities IT function shadowing for disaster recovery Virtual enterprise, collaboratory, and community Our Goal To build a value-added application service hosting platform based on shared infrastructure, achieving: On-demand creation and provisioning Virtualization Isolation Protection Accountability Privacy Related Work Utility computing architectures VERITAS, HP UDC, IBM Oceano Grid platforms Computation: Globus, Condor, Legion, NetSolve, Harness, Cactus Storage and data: SRB, NeST, Data Grid, OceanStore Shared infrastructure PlanetLab, Emulab Active services Active Service Grid, Berkeley Active Service Framework, CANS (NYU), Darwin, WebOS Related Work Resource isolation GARA, QLinux (UMass), Virtual service (UMich), Resource Container, Cluster Reserves (Rice) Virtualization technologies Virtual super computer (aggregation): NOW, HPVM Virtual OS, isolation kernel (slicing): VMWare, Xen (Cambridge), Denali (UW), UML, UMLinux, Virtual Private Server (Ensim) Grid computing on VM: Virtuoso (Northwestern), Entropia Virtual cluster: Cluster-on-Demand (Duke) SODA Service-On-Demand Architecture for application service hosting utility platforms Research components of SODA General architecture Protection, intrusion detection, logging Confined and VM-based overlay Market-driven planning and management Outline Research components of SODA: General architecture Security and protection Confined VM-based overlay ‘Property’ planning and management Detailed Information Xuxian Jiang, Dongyan Xu, "SODA: a Service-On-Demand Architecture for Application Service Hosting Utility Platforms", Proceedings of The 12th IEEE International Symposium on High Performance Distributed Computing (HPDC-12), Seattle, WA, June 2003. Overview of SODA AS Virtual service node AS’ SODA Host (physical) Virtualization: Key Technique Two-level OS structure Host OS Guest OS Strong isolation Administration isolation Installation isolation Fault / attack Isolation Recovery, migration, and forensics Virtual service node Application service (AS) Guest OS Internetworking enabled AS1 ASn … Guest OS Guest OS Host OS One SODA host Service Requests From Clients Service Requests From Clients Service Switch for S Virtual service node Service S Guest OS SODA Daemon Host OS SODA Master Service Switch for S’ Service S Service S’ Guest OS Guest OS SODA Daemon Service S’ Guest OS Host OS SODA Daemon Host OS SODA Agent Service Creation Requests From ASP On the Same SODA Host WWW service Honeypot Host OS and Guest OS Guest OS: based on User-Mode Linux (UML), an open-source virtual OS (different from UMLinux and VServer ) By Jeff Dike, http://user-mode-linux.sourceforge.net Running in user space of host OS Separate kernel address space Physical memory usage limit Host OS: Linux (linux-2.4.19, enhanced) CPU fair share scheduler (for CPU isolation between virtual service nodes) Experiment: CPU Isolation VM1: CPU-intensive Original Linux Scheduler VM2: IO-intensive VM3: Web Enhanced Linux Scheduler On-Demand Service Priming Performed by SODA Daemon Customization of guest OS (“cook to order” ) Active service image downloading Automatic bootstrapping of virtual service node Service Bootstrapping Time Linux Configuration Image size Time (seattle) Time (tacoma) Rootfs_tomrtbt_ 1.7.205 15 MB 2.0 sec. 3.0 sec. Rootfs_base_1.0 29.3 MB 3.0 sec. 4.0 sec. Root_fs_lfs_4.0 400 MB 4.0 sec. 16.0 sec. Root_fs.rh-7.2server.pristine.200 21012 253 MB 22.0 sec. 42.0 sec. Slow-Down (w/o optimization) System call level (clock cycles) System call UML Linux getpid 27,276 1,208 geteuid 26,648 1,064 dup2 26,904 1,084 mmap 27,864 1,208 munmap 27,044 1,200 gettimeofday 37,004 1,368 Application level Outline Research components of SODA: General architecture Security and protection Confined VM-based overlay ‘Property’ planning and management Detailed Information Xuxian Jiang, Dongyan Xu, Rudolf Eigenmann, "Protection Mechanisms for Application Service Hosting Platforms", Proceedings of IEEE/ACM Int'l Symposium on Cluster Computing and the Grid (CCGrid 2004), Chicago, IL, April 2004. Xuxian Jiang, Dongyan Xu, "Collapsar: A VM-Based Architecture for Network Attack Detention Center", to appear in Proceedings of the 13th USENIX Security Symposium (Security '04), San Diego, CA, August 2004. Security and Protection Virtual switching and firewalling IDS in guest OS kernel Untamperable logging (‘blackbox’-ing) AS1 ASn … Guest OS Host OS Guest OS Virtual Switching and Firewalling Virtual machine (with IP addr.) Guest OS Host OS SODA host (Invisible on Internet) Guest OS Guest OS Firewall Kernort: IDS in Guest OS Kernel Problems with traditional IDS Encrypted traffic (e.g. ssh) makes NIDS less effective App-level IDS process will be “killed”, once a machine is compromised Log may be tampered with Fail-open Related projects Backtracker (Michigan) VMM-based retrospection (Stanford) Forensix (OHSU) ESP (Purdue CERIAS) Open-source projects: Snort, Saint Jude Kernort VM-based IDS Deployed in each VM Inside guest OS kernel: a unique vista point Customizable without affecting host OS Clearer view Untamperable logging (saved to SODA host) Renewable signature (read from SODA host) Fail-close instead of fail-open Kernort: IDS in Guest OS Kernel Guest OS Guest OS Kernort Components Kernort sensor Event-driven (system call and packet reception) Renewable signature set Matching against a small signature set (“Top 20 most wanted”) Kernort blackbox Untamperable logging Privacy preservation of ASes Analyzer Exhaustive signature matching Detection of complex attack patterns Session replay Kernort Virtual machine Host OS Kernort (shaded areas: logs) Real-Time Alert Session Re-play Impact on Performance Impact on Performance Outline Research components of SODA: General architecture Security and protection Confined VM-based overlay ‘Property’ planning and management Detailed Information Xuxian Jiang, Dongyan Xu, "vBET: a VM-Based Emulation Testbed", Proceedings of ACM Workshop on Models, Methods and Tools for Reproducible Network Research (MoMeTools, in conjunction with ACM SIGCOMM 2003), Karlsruhe, Germany, August 2003. Xuxian Jiang, Dongyan Xu, "VIOLIN: Virtual Internetworking on OverLay INfrastructure", Department of Computer Sciences Technical Report CSD TR 03-027, Purdue University, July 2003. Xuxian Jiang, Dongyan Xu, “A Middleware Architecture for Confined Virtual Machine Overlays", in preparation, March 2004. Traditional Overlay Network Problems with traditional overlays: Open for attacks Attacks from the outside (i.e. Internet) against overlay nodes Attacks from an overlay node against the outside Difficult to manage An overlay across multiple administration domains A host participate in multiple overlays Difficult to enforce overlay topology and traffic volume VPN does not solve the problems Traditional Overlay Network Firewall Firewall Firewall VM-based Overlay The case for VM-based overlay Multiple overlays on shared infrastructure On-demand creation Confinement and isolation VM introduces new network administration complexity “What is this new machine that has suddenly appeared in my domain?” “Where is the machine that was in my domain yesterday?” “How much network connectivity should a VM have?” “How many IP addresses for VMs?” Confined VM-based Overlay In addition to VM, we need VN for VMs VN: a highly overloaded term (VPN, X-bone…) What is new: Confined and VM-based overlays Applications Multi-institutional collaborations Philanthropic (volunteer) computing systems Network emulations Confined VM-based Overlay VM VM VM ≤2Mbps ≤2Mbps Virtual infrastructure ≤1Mbps Firewall Firewall Firewall Key Properties Confined overlay topology and traffic No attack possible from inside the overlay to the outside world Virtual IP address space No need for application modification and re-compilation A More Generic Picture VIOLIN: Virtual Internetworking on OverLay INfrastructure vBET: an Example of Confined Overlays on Demand An education tool for network and distributed system emulation Fidelity-preserving setup Maneuverable network entities Real-world network software Strict confinement (network security experiment) Flexible configuration Not constrained by device/port availability No manual cable re-wiring or hardware setup Simultaneous experiments Cost-effective vBET Features Can be deployed in n ≥ 1 vBET servers Efficient startup and tear-down of emulated entities Strong network virtualization IP address space Virtual routers, switches, firewalls, end-hosts, links Communications confined by virtual topology Dynamic addition, deletion, migration, configuration of network entities vBET GUI Sample Emulation: OSPF Routing Emulation of OSPF Routing Demo video clip at: http://www.cs.purdue.edu/~jiangx/vBET/videos/vbet_ospf.avi Sample Emulation: Distributed Firewalls Screenshot Sample Emulation: Chord P2P Network Screenshot Outline Research components of SODA: General architecture Security and protection Confined VM-based overlay ‘Property’ planning and management Property Planning and Management Tenant selection: Among a set of potential tenants (ASes), which ones to host? (for maximum revenue, resource utilization, security…) SODA provider selection: Among a set of SODA providers, which one should be chosen to host an AS? Property Planning and Management Examples of bad planning: Many PDA transcoding ASes in an area with a small PDA user population AS not requiring client registration and log-in (potential DDoS attacks) Majority of ASes exhibiting similar demand characteristics such as: Load Time Property Planning and Management AS profiling Resource requirement Security/authentication Demand characteristics Market analysis Competing ASes, market size/growth/expected share ASes correlation (“80% of clients requesting AS X also request AS Y” ) Trading/pricing of SODA machine slices Property Planning and Management Forming alliance of SODA providers Property Planning and Management Forming alliance of SODA providers Summary Virtualization: a key enabling technology in realizing utility computing vision Hosting utility is more complex than computation utility (host – tenants – clients) SODA achieves: On-demand service creation Service virtualization, isolation and confinement Protection, accountability, privacy Overlay isolation and confinement Ongoing Work VM/service migration, shadowing, recovery Service profiling, accounting, auditing (resources, security) Market-driven planning, provisioning, and management (SODA ecology) Deployment and evaluation (Purdue Bindley Bioscience Center) Thank you. For more information: {dxu, jiangx}@cs.purdue.edu http://www.cs.purdue.edu/~dxu AOL keywords “Purdue SODA Friends”