Ally: OS-Transparent Packet Inspection Using Sequestered Cores

advertisement
Ally: OS-Transparent Packet Inspection
Using Sequestered Cores
Jen-Cheng Huang 1, Matteo Monchiero2, Yoshio Turner3, Hsien-Hsin Lee1
1Georgia
Tech
2Intel
Labs
3HP
Labs
1
Deep Packet Inspection (DPI)
Deployment of Packet Processing Services
Middle
Boxes
Internet
Data Center
Intrusion
Detection
Content
Insertion
Traffic
Classification
2
Problem
Local Traffic is growing in importance…
But The traffic within the data center is not inspected!
Internet
Middle
Boxes
Data Center
3
Approach
“Co-locate” DPI with the server
DPI appliance
Leverage abundant
CPU resources
Leverage existing
management interfaces
on servers, e.g. HP iLO
Server
Compatible with
heterogeneous architecture,
e.g. on-chip accelerators
4
Requirements
• Transparency
– Independent to the server’s software stack
• Efficiency
– Low overhead packet interception
• Isolation
– Resistant to attacks
5
Related Work
ETTM:
a scalableSupport
fault tolerant
manager. C. Dixon et al. NSDI ‘11
Virtualization
for network
DPI deployment
Transparency
Hypervisor Overhead
Guest
VM
SW
Hypervisor
HW
Processors
DPI
VM
Hypervisor Vulnerability
Virtualized Platform
6
Ally Architecture
Multi-core processor
Unprivileged Partition
Software Stack
(OS +
Applications)
core
core
core
Privileged Partition
NIC
Traffic
core
Software Stack
(DPI
Application)
core
core
NIC
7
Outline
• Introduction & Motivation
• Architecture
– Overview
– Multicore Partitioning
– Packet interception
• Evaluation
• Conclusions
8
MMU
MMU
Last
Level
Cache
MMU
Main
Memory
Core
MMU
MMU
Interrupt
Controller
Core
MMU
Interrupt
Controller
Baseline Architecture
Interconnect
Northbridge
Memory
Controller
NIC
External
Network
Interrupt
Unit
IOMMU
Service
Processor
BIOS
Management
Network
9
Ally Architecture
MMU
MMU
Last
Level
Cache
MMU
Main
Memory
Core
MMU
MMU
Interrupt
Controller
Core
Privileged partition
MMU
Interrupt
Controller
Unprivileged partition
Interconnect
Northbridge
Memory
Controller
NIC
External
Network
Interrupt
Unit
IOMMU
Service
Processor
BIOS
Management
Network
10
Outline
• Introduction & Motivation
• Architecture
– Overview
– Multicore Partitioning
– Packet interception
• Evaluation
• Conclusions
11
Multicore Partitioning
Multi-core processor
Unprivileged Partition
Privileged Partition
Invisible
Software Stack
(OS +
Applications)
core
core
core
Software Stack
(DPI Application)
core
core
core
NIC
12
Core Sequestration
Modify the BIOS to hide privileged core information from the OS
Ally Booting Procedure:
…...
 BSP core - the first core that boots
 AP cores - the other cores
 IPI - Inter-processor interrupts
AP
Initialize
DPI
Engine
DPI core waits for
IN/OUT packets
AP
BSP
Core Info
Table
AP
Wakeup
IPI
OS retrieves
cores information
Update
AP
13
Memory Protection
Partition the memory into two physically contiguous regions
Privileged
partition
Page Table
CR3
Unprivileged
partition
Boundary
Register
Main
Memory
TLB Miss Handler
TLB
TLB
Miss
Range Checking
TLB Fill
MMU
Unprivileged Core
14
Outline
• Introduction & Motivation
• Architecture
– Overview
– Multicore Partitioning
– Packet interception
• Evaluation
• Conclusions
15
Packet Interception
Multi-core processor
Unprivileged Partition
Software Stack
(OS +
Applications)
core
core
core
Privileged Partition
NIC
Traffic
core
Software Stack
(DPI Application)
core
core
NIC
16
Packet Interception
Virtualization of the Descriptor Queues
OS memory
DPI memory
Descriptor queues
Only one copy of the packet
buffers
Descriptor queues replicated
NIC
17
Packet Interception
• Virtualization of the Descriptor Queues
– Device independent, software independent
– No copying on packet buffers
• Processor and NIC communication
– Queue manipulation uses Memory Mapped IO (MMIO)
accesses
– NIC event notification uses Interrupt
18
MMIO redirection
OS
core
IPI
Load/store
MMU
MMU detects
specific MMIO
addresses
R/W redirection
MMU redirects RW
to a reserved region
in DPI memory
DPI
core
DPI
memory
MMU sends IPI to DPI core
19
Ally Hardware Properties
• Simple extensions to existing hardware
components
• No impact expected on critical timing paths
• Compatible with virtualization support (Intel VTx/EPT, AMD SVM/NPT)
20
Outline
• Introduction & Motivation
• Architecture
– Overview
– Multicore Partitioning
– Packet interception
• Evaluation
• Conclusions
21
Evaluation
Full system emulation
QEMU
Core sequestration
HW changes
Real machine prototype
Hardware
– Intel Core 2 duo 2.66 GHz with 1 Gbit Intel NIC
Benchmarks
– Netperf
– SPECweb
Systems
– Ally, Linux and Xen
22
System Configurations
Netperf/
Specweb
Netperf/
Specweb
Snort
Snort
Kernel
Kernel
NIC Driver
SW
HW
Queue
Virtualization
SW
OS core
DPI core
Ally
IP queue
HW
NIC Driver
OS core
DPI core
Linux
23
System Configurations
Netperf/
Specweb
Snort
Dom0
Kernel
SW
HW
DomU
Kernel
Hypervisor
OS core
DPI core
Xen
24
Netperf CPU Usage
25
cycles/request * 106
SPECweb CPU Usage
26
Outline
• Introduction & Motivation
• Architecture
– Overview
– Multicore Partitioning
– Packet interception
• Evaluation
• Conclusions
27
Conclusions
 Ally: a framework for transparent deployment of
packet inspection appliances
 Ally uses a set of simple HW/FW extensions
enable reliable multicore partitioning and efficient
packet inspection
 Ally is fully compatible with new virtualization
technology as well as heterogeneous architecture
28
Thanks
29
Throughput
30
DPI using Network Processor
31
Conventional Architecture
Multi-core processor
Unprivileged Partition
Software Stack
(OS + Applications)
core
core
core
cores
cores
cores
NIC
32
Transmission Path
Multi-core processor
Unprivileged Partition
Software Stack
(OS +
Applications)
core
core
core
Privileged Partition
Software Stack
(DPI
Application)
core
core
core
NIC
33
Receive Path
Multi-core processor
Unprivileged Partition
Software Stack
(OS + Applications)
core
core
core
Privileged Partition
Software Stack
(DPI Application)
core
core
core
NIC
34
Unprivileged partition
Privileged
partition
Unprivilege
d partition
MMU
MMU
MMU
Last
Level
Cache
DPI core
DPI core
DPI core
Local APIC
Local APIC
Local APIC
OS core
OS core
OS core
Privileged partition
MMU
MMU
MMU
Interface
Interface
Interface
Main
Memory
Local APIC
Local APIC
Local APIC
Processor
Interface
Interface
Interface
On chip interconnect
Integrated Northbridge
Memory
Controller
PCIe ctrl
DMI Ctrl
IOMMU
Interrupt
Unit
Platform Controller Hub
Network
Management
Network
NIC
Managemen
t NIC
IOAPIC
Service
Processor
BIOS
35
Unprivileged partition
Privileged
partition
Unprivilege
d partition
MMU
MMU
MMU
Last
Level
Cache
DPI core
DPI core
DPI core
Local APIC
Local APIC
Local APIC
OS core
OS core
OS core
Privileged partition
MMU
MMU
MMU
Interface
Interface
Interface
Main
Memory
Local APIC
Local APIC
Local APIC
Processor
Interface
Interface
Interface
On chip interconnect
Integrated Northbridge
Memory
Controller
PCIe ctrl
DMI Ctrl
IOMMU
Interrupt
Unit
Platform Controller Hub
Network
Management
Network
NIC
Managemen
t NIC
IOAPIC
Service
Processor
BIOS
36
MMU Modification – Memory Protection
DPI core
boundary register
Page Table
Privileged
partition
Unprivilege
d partition
CR3
Special_reg
TLB Miss Handler
TLB
phys_addr
>
special_reg ?
Main
Memory
37
Memory Protection Procedure
DPI core
boundary register
Page Table
Privileged
partition
Unprivilege
d partition
CR3
TLB miss
Virtual
Address
TLB
Special_reg
TLB Miss Handler
phys_addr
>
special_reg ?
Main
Memory
38
Memory Protection Procedure
DPI core
boundary register
Page Table
Privileged
partition
Unprivilege
d partition
CR3
TLB miss
Virtual
Address
Special_reg
TLB Miss Handler
phys_addr
>
special_reg ?
TLB
Main
Memory
TLB fill
39
Memory Protection
Multi-core processor
Unprivileged Partition
Invisible
Software Stack
(OS +
Applications)
core
core
core
Privileged Partition
core
Software Stack
(DPI Application)
core
core
NIC
40
Unprivileged partition
Privileged
partition
Unprivilege
d partition
MMU
MMU
MMU
Last
Level
Cache
DPI core
DPI core
DPI core
Local APIC
Local APIC
Local APIC
OS core
OS core
OS core
Privileged partition
MMU
MMU
MMU
Interface
Interface
Interface
Main
Memory
Local APIC
Local APIC
Local APIC
Processor
Interface
Interface
Interface
On chip interconnect
Integrated Northbridge
Memory
Controller
PCIe ctrl
DMI Ctrl
IOMMU
Interrupt
Unit
Platform Controller Hub
Network
Management
Network
NIC
Management
NIC
IOAPIC
Service
Processor
BIOS
41
Unprivileged partition
Privileged
partition
Unprivilege
d partition
MMU
MMU
MMU
Last
Level
Cache
DPI core
DPI core
DPI core
Local APIC
Local APIC
Local APIC
OS core
OS core
OS core
Privileged partition
MMU
MMU
MMU
Interface
Interface
Interface
Main
Memory
Local APIC
Local APIC
Local APIC
Processor
Interface
Interface
Interface
On chip interconnect
Integrated Northbridge
Memory
Controller
PCIe ctrl
DMI Ctrl
IOMMU
Interrupt
Unit
Platform Controller Hub
Network
Management
Network
NIC
Managemen
t NIC
IOAPIC
Service
Processor
BIOS
42
MMU Modification – MMIO Redirection
Physical Page Redirection Bit
TLB Miss Handler
TLB
Check uncacheable
address map
Physical
Address
Remapped
Address
Redirection
Table
43
MMIO Redirection – TLB Miss
Physical Page Redirection Bit
TLB miss
TLB Miss Handler
Page Table Lookup
TLB
Virtual
Address
• On a TLB miss, the TLB miss handler does the
page table walk
44
MMIO Redirection – TLB Miss
Physical Page Redirection Bit
TLB Miss Handler
TLB
Check uncacheable
address map
Physical
Address
• The TMH checks if the resulting physical address
falls in an uncacheable page and hence
potentially a MMIO page
45
MMIO Redirection – TLB Miss
Physical Page Redirection Bit
TLB Miss Handler
Check uncacheable
address map
TLB
Physical
Address
Physical
Address
Redirection
Table
Remapped
Address
• If the page is uncacheable, the TMH looks up the
redirection table to check if any address in this
page needs to be redirected
46
MMIO Redirection – TLB Miss
Physical Page Redirection Bit
TLB Miss Handler
TLB fill
TLB
Physical
Address
Check uncacheable
address map
Redirection
Table
Remapped
Address
• If any address in the page needs to be redirected,
the TMH sets the redirection bit in addition to fill
the TLB
47
MMIO Redirection – TLB Hit
Redirection Bit
Physical Page
TLB
Virtual
Address
Physical
Address
Remapped
Address
Physical
Address
Offset
LLC
• On a TLB hit, if the redirection bit is set, the MMU
looks up the Last Level Cache (LLC) used to
cache translations in Redirection Table
48
MMIO Redirection – TLB Hit
Redirection Bit
Physical Page
TLB
Physical
Address
Physical
Address
Remapped
Address
Translated
Address
Hit
LLC
Generate IPI
• If a translation is found, the MMU returns the
translated address and sends IPI to privileged
cores.
49
MMIO Redirection – TLB Hit
Redirection Bit
Physical Page
TLB
Physical
Address
Physical
Address
Remapped
Address
Miss
Redirection Table
Lookup
LLC
• If the LLC misses, then Redirection Table Lookup
is performed
50
Interrupt Unit Modification
DPI
core
OS
core
If Source == NIC,
Redirect Interrupt
Interrupt Unit
NIC
51
Interrupt Redirection
DPI
core
OS
core
If Source == NIC,
Redirect Interrupt
Interrupt Unit
Interrupt
NIC
• When NIC raises an interrupt, The interrupt Unit
redirects the interrupt to DPI core
52
Interrupt Redirection
DPI
core
OS
core
IPI
If Source == NIC,
Redirect Interrupt
Interrupt Unit
NIC
• After the NIC interrupt is handled, DPI core sends
an IPI to OS core mimicking NIC interrupt
53
Summary of Hardware Modifications
Unit
Description
Purpose
OS-core
MMU
Prevent memory accesses to DPI memory from OS- Protection
core
Redirect MMIO accesses to DPI memory from OScore and interrupt DPI core
Packet
Interception
IOMMU
Prevent non authorized DMA to DPI Memory
Protection
IOAPIC
Redirect NIC interrupts to DPI-core
Packet
Interception
All Units
Protected configuration registers
Protection
54
Functional Evaluation
Full system emulation
• QEMU
• Validate Hardware and Firmware Changes
55
DPI core Usage
56
SPECweb Cache Misses
57
Memory Protection
Multi-core processor
Unprivileged Partition
Invisible
Software Stack
(OS +
Applications)
core
core
core
How?
Modified MMU
Privileged Partition
core
Software Stack
(DPI Application)
core
core
NIC
58
Challenges
- Make privileged partition protected and invisible
from the unprivileged partition
- Core Sequestration
- Memory Protection
- Intercept packets efficiently
- Packet Interception
59
Ally System
Linux
Core
Other
Apps
kernel
DPI
Core
Snort
NIC Driver
Queue
Virtualization
NIC
Traffic
NIC
60
Linux System
Linux
Core
Core
Snort
Other
Apps
kernel
IP queue
NIC Driver
NIC
Traffic
NIC
61
Xen System
Linux
Core
Core
Snort
Other
Apps
VM #0
VM #1
IP
queue
Hypervisor
NIC
Traffic
NIC
62
Download
Related flashcards
Create Flashcards