Ally: OS-Transparent Packet Inspection Using Sequestered Cores
advertisement
Ally: OS-Transparent Packet Inspection Using Sequestered Cores Jen-Cheng Huang 1, Matteo Monchiero2, Yoshio Turner3, Hsien-Hsin Lee1 1Georgia Tech 2Intel Labs 3HP Labs 1 Deep Packet Inspection (DPI) Deployment of Packet Processing Services Middle Boxes Internet Data Center Intrusion Detection Content Insertion Traffic Classification 2 Problem Local Traffic is growing in importance… But The traffic within the data center is not inspected! Internet Middle Boxes Data Center 3 Approach “Co-locate” DPI with the server DPI appliance Leverage abundant CPU resources Leverage existing management interfaces on servers, e.g. HP iLO Server Compatible with heterogeneous architecture, e.g. on-chip accelerators 4 Requirements • Transparency – Independent to the server’s software stack • Efficiency – Low overhead packet interception • Isolation – Resistant to attacks 5 Related Work ETTM: a scalableSupport fault tolerant manager. C. Dixon et al. NSDI ‘11 Virtualization for network DPI deployment Transparency Hypervisor Overhead Guest VM SW Hypervisor HW Processors DPI VM Hypervisor Vulnerability Virtualized Platform 6 Ally Architecture Multi-core processor Unprivileged Partition Software Stack (OS + Applications) core core core Privileged Partition NIC Traffic core Software Stack (DPI Application) core core NIC 7 Outline • Introduction & Motivation • Architecture – Overview – Multicore Partitioning – Packet interception • Evaluation • Conclusions 8 MMU MMU Last Level Cache MMU Main Memory Core MMU MMU Interrupt Controller Core MMU Interrupt Controller Baseline Architecture Interconnect Northbridge Memory Controller NIC External Network Interrupt Unit IOMMU Service Processor BIOS Management Network 9 Ally Architecture MMU MMU Last Level Cache MMU Main Memory Core MMU MMU Interrupt Controller Core Privileged partition MMU Interrupt Controller Unprivileged partition Interconnect Northbridge Memory Controller NIC External Network Interrupt Unit IOMMU Service Processor BIOS Management Network 10 Outline • Introduction & Motivation • Architecture – Overview – Multicore Partitioning – Packet interception • Evaluation • Conclusions 11 Multicore Partitioning Multi-core processor Unprivileged Partition Privileged Partition Invisible Software Stack (OS + Applications) core core core Software Stack (DPI Application) core core core NIC 12 Core Sequestration Modify the BIOS to hide privileged core information from the OS Ally Booting Procedure: …... BSP core - the first core that boots AP cores - the other cores IPI - Inter-processor interrupts AP Initialize DPI Engine DPI core waits for IN/OUT packets AP BSP Core Info Table AP Wakeup IPI OS retrieves cores information Update AP 13 Memory Protection Partition the memory into two physically contiguous regions Privileged partition Page Table CR3 Unprivileged partition Boundary Register Main Memory TLB Miss Handler TLB TLB Miss Range Checking TLB Fill MMU Unprivileged Core 14 Outline • Introduction & Motivation • Architecture – Overview – Multicore Partitioning – Packet interception • Evaluation • Conclusions 15 Packet Interception Multi-core processor Unprivileged Partition Software Stack (OS + Applications) core core core Privileged Partition NIC Traffic core Software Stack (DPI Application) core core NIC 16 Packet Interception Virtualization of the Descriptor Queues OS memory DPI memory Descriptor queues Only one copy of the packet buffers Descriptor queues replicated NIC 17 Packet Interception • Virtualization of the Descriptor Queues – Device independent, software independent – No copying on packet buffers • Processor and NIC communication – Queue manipulation uses Memory Mapped IO (MMIO) accesses – NIC event notification uses Interrupt 18 MMIO redirection OS core IPI Load/store MMU MMU detects specific MMIO addresses R/W redirection MMU redirects RW to a reserved region in DPI memory DPI core DPI memory MMU sends IPI to DPI core 19 Ally Hardware Properties • Simple extensions to existing hardware components • No impact expected on critical timing paths • Compatible with virtualization support (Intel VTx/EPT, AMD SVM/NPT) 20 Outline • Introduction & Motivation • Architecture – Overview – Multicore Partitioning – Packet interception • Evaluation • Conclusions 21 Evaluation Full system emulation QEMU Core sequestration HW changes Real machine prototype Hardware – Intel Core 2 duo 2.66 GHz with 1 Gbit Intel NIC Benchmarks – Netperf – SPECweb Systems – Ally, Linux and Xen 22 System Configurations Netperf/ Specweb Netperf/ Specweb Snort Snort Kernel Kernel NIC Driver SW HW Queue Virtualization SW OS core DPI core Ally IP queue HW NIC Driver OS core DPI core Linux 23 System Configurations Netperf/ Specweb Snort Dom0 Kernel SW HW DomU Kernel Hypervisor OS core DPI core Xen 24 Netperf CPU Usage 25 cycles/request * 106 SPECweb CPU Usage 26 Outline • Introduction & Motivation • Architecture – Overview – Multicore Partitioning – Packet interception • Evaluation • Conclusions 27 Conclusions Ally: a framework for transparent deployment of packet inspection appliances Ally uses a set of simple HW/FW extensions enable reliable multicore partitioning and efficient packet inspection Ally is fully compatible with new virtualization technology as well as heterogeneous architecture 28 Thanks 29 Throughput 30 DPI using Network Processor 31 Conventional Architecture Multi-core processor Unprivileged Partition Software Stack (OS + Applications) core core core cores cores cores NIC 32 Transmission Path Multi-core processor Unprivileged Partition Software Stack (OS + Applications) core core core Privileged Partition Software Stack (DPI Application) core core core NIC 33 Receive Path Multi-core processor Unprivileged Partition Software Stack (OS + Applications) core core core Privileged Partition Software Stack (DPI Application) core core core NIC 34 Unprivileged partition Privileged partition Unprivilege d partition MMU MMU MMU Last Level Cache DPI core DPI core DPI core Local APIC Local APIC Local APIC OS core OS core OS core Privileged partition MMU MMU MMU Interface Interface Interface Main Memory Local APIC Local APIC Local APIC Processor Interface Interface Interface On chip interconnect Integrated Northbridge Memory Controller PCIe ctrl DMI Ctrl IOMMU Interrupt Unit Platform Controller Hub Network Management Network NIC Managemen t NIC IOAPIC Service Processor BIOS 35 Unprivileged partition Privileged partition Unprivilege d partition MMU MMU MMU Last Level Cache DPI core DPI core DPI core Local APIC Local APIC Local APIC OS core OS core OS core Privileged partition MMU MMU MMU Interface Interface Interface Main Memory Local APIC Local APIC Local APIC Processor Interface Interface Interface On chip interconnect Integrated Northbridge Memory Controller PCIe ctrl DMI Ctrl IOMMU Interrupt Unit Platform Controller Hub Network Management Network NIC Managemen t NIC IOAPIC Service Processor BIOS 36 MMU Modification – Memory Protection DPI core boundary register Page Table Privileged partition Unprivilege d partition CR3 Special_reg TLB Miss Handler TLB phys_addr > special_reg ? Main Memory 37 Memory Protection Procedure DPI core boundary register Page Table Privileged partition Unprivilege d partition CR3 TLB miss Virtual Address TLB Special_reg TLB Miss Handler phys_addr > special_reg ? Main Memory 38 Memory Protection Procedure DPI core boundary register Page Table Privileged partition Unprivilege d partition CR3 TLB miss Virtual Address Special_reg TLB Miss Handler phys_addr > special_reg ? TLB Main Memory TLB fill 39 Memory Protection Multi-core processor Unprivileged Partition Invisible Software Stack (OS + Applications) core core core Privileged Partition core Software Stack (DPI Application) core core NIC 40 Unprivileged partition Privileged partition Unprivilege d partition MMU MMU MMU Last Level Cache DPI core DPI core DPI core Local APIC Local APIC Local APIC OS core OS core OS core Privileged partition MMU MMU MMU Interface Interface Interface Main Memory Local APIC Local APIC Local APIC Processor Interface Interface Interface On chip interconnect Integrated Northbridge Memory Controller PCIe ctrl DMI Ctrl IOMMU Interrupt Unit Platform Controller Hub Network Management Network NIC Management NIC IOAPIC Service Processor BIOS 41 Unprivileged partition Privileged partition Unprivilege d partition MMU MMU MMU Last Level Cache DPI core DPI core DPI core Local APIC Local APIC Local APIC OS core OS core OS core Privileged partition MMU MMU MMU Interface Interface Interface Main Memory Local APIC Local APIC Local APIC Processor Interface Interface Interface On chip interconnect Integrated Northbridge Memory Controller PCIe ctrl DMI Ctrl IOMMU Interrupt Unit Platform Controller Hub Network Management Network NIC Managemen t NIC IOAPIC Service Processor BIOS 42 MMU Modification – MMIO Redirection Physical Page Redirection Bit TLB Miss Handler TLB Check uncacheable address map Physical Address Remapped Address Redirection Table 43 MMIO Redirection – TLB Miss Physical Page Redirection Bit TLB miss TLB Miss Handler Page Table Lookup TLB Virtual Address • On a TLB miss, the TLB miss handler does the page table walk 44 MMIO Redirection – TLB Miss Physical Page Redirection Bit TLB Miss Handler TLB Check uncacheable address map Physical Address • The TMH checks if the resulting physical address falls in an uncacheable page and hence potentially a MMIO page 45 MMIO Redirection – TLB Miss Physical Page Redirection Bit TLB Miss Handler Check uncacheable address map TLB Physical Address Physical Address Redirection Table Remapped Address • If the page is uncacheable, the TMH looks up the redirection table to check if any address in this page needs to be redirected 46 MMIO Redirection – TLB Miss Physical Page Redirection Bit TLB Miss Handler TLB fill TLB Physical Address Check uncacheable address map Redirection Table Remapped Address • If any address in the page needs to be redirected, the TMH sets the redirection bit in addition to fill the TLB 47 MMIO Redirection – TLB Hit Redirection Bit Physical Page TLB Virtual Address Physical Address Remapped Address Physical Address Offset LLC • On a TLB hit, if the redirection bit is set, the MMU looks up the Last Level Cache (LLC) used to cache translations in Redirection Table 48 MMIO Redirection – TLB Hit Redirection Bit Physical Page TLB Physical Address Physical Address Remapped Address Translated Address Hit LLC Generate IPI • If a translation is found, the MMU returns the translated address and sends IPI to privileged cores. 49 MMIO Redirection – TLB Hit Redirection Bit Physical Page TLB Physical Address Physical Address Remapped Address Miss Redirection Table Lookup LLC • If the LLC misses, then Redirection Table Lookup is performed 50 Interrupt Unit Modification DPI core OS core If Source == NIC, Redirect Interrupt Interrupt Unit NIC 51 Interrupt Redirection DPI core OS core If Source == NIC, Redirect Interrupt Interrupt Unit Interrupt NIC • When NIC raises an interrupt, The interrupt Unit redirects the interrupt to DPI core 52 Interrupt Redirection DPI core OS core IPI If Source == NIC, Redirect Interrupt Interrupt Unit NIC • After the NIC interrupt is handled, DPI core sends an IPI to OS core mimicking NIC interrupt 53 Summary of Hardware Modifications Unit Description Purpose OS-core MMU Prevent memory accesses to DPI memory from OS- Protection core Redirect MMIO accesses to DPI memory from OScore and interrupt DPI core Packet Interception IOMMU Prevent non authorized DMA to DPI Memory Protection IOAPIC Redirect NIC interrupts to DPI-core Packet Interception All Units Protected configuration registers Protection 54 Functional Evaluation Full system emulation • QEMU • Validate Hardware and Firmware Changes 55 DPI core Usage 56 SPECweb Cache Misses 57 Memory Protection Multi-core processor Unprivileged Partition Invisible Software Stack (OS + Applications) core core core How? Modified MMU Privileged Partition core Software Stack (DPI Application) core core NIC 58 Challenges - Make privileged partition protected and invisible from the unprivileged partition - Core Sequestration - Memory Protection - Intercept packets efficiently - Packet Interception 59 Ally System Linux Core Other Apps kernel DPI Core Snort NIC Driver Queue Virtualization NIC Traffic NIC 60 Linux System Linux Core Core Snort Other Apps kernel IP queue NIC Driver NIC Traffic NIC 61 Xen System Linux Core Core Snort Other Apps VM #0 VM #1 IP queue Hypervisor NIC Traffic NIC 62
