Computer Science
Cloud Computing Infrastructure Security
Peng Ning
With Ahmed Azab, Xiaolan Zhang, Wu Zhou, Xuxian Jiang, and Zhi Wang.
Supported by the US NSF under grant # 0910767, the US ARO under the grant # W911NF-08-1-0105,
and IBM under Open Collaboration Research (OCR) Awards.
June 29, 2012
ACNS 2012
1
Outline
• Background
• Security threats to cloud computing
• Security of cloud computing infrastructure
–
–
–
–
–
Driven by a new security architecture for cloud computing
Hypervisor-based security services
Offline VM image security services
Hypervisor integrity services
Isolation that can bypass hypervisor control
• Conclusion
Computer Science
June 29, 2012
ACNS 2012
2
What is Cloud Computing
• Wikipedia
– Cloud computing is a paradigm of computing in which
dynamically scalable and often virtualized resources are
provided as a service over the Internet
– Users need not have knowledge of, expertise in, or control
over the technology infrastructure in the "cloud" that
supports them
– Reduction in hardware, operational and administrative costs
• Virtualization is a key to cloud computing
– Scalability
– Ease of use
– Affordable pricing
Computer Science
June 29, 2012
ACNS 2012
3
Example: Amazon Elastic Compute Cloud (EC2)
EC 2Start
Management
Console
Set
theVerify
Instance
Instance
Launch
thethe
Instance
an up
Instance
Computer Science
June 29, 2012
ACNS 2012
4
Security Threats in Cloud Computing
External threats
Guest-to-guest threats
Guest-to-cloud threats
Cloud-to-guest threats
Computer Science
June 29, 2012
ACNS 2012
5
Cloud Computing Infrastructure Security
• Our proposal
– A security architecture for compute clouds
• Focus on Infrastructure as a Service (IaaS)
– Addition of security architecture components
•
•
•
•
Hypervisor-based security services
Offline VM image security services
Hypervisor integrity services
Isolation mechanisms that can bypass the hypervisor
Computer Science
June 29, 2012
ACNS 2012
6
A Typical Compute Cloud
Computer Science
June 29, 2012
ACNS 2012
7
Virtualization-based Runtime Security Services
HIMA [ACSAC ’09]
HookSafe [CCS ’09]
Computer Science
June 29, 2012
ACNS 2012
8
Example Service: HIMA
• HIMA: Hypervisor based Integrity Measurement Agent
• Validation of VMs with runtime guarantees
– Measure the VM OS and applications loaded into guest VMs
– Actively monitor all guest events that could change measured
applications
• Time of Check to Time of Use (TOCTTOU) consistency
• Prototypes
– Initial implementation works for Xen (para-virtualization)
– Ported to support KVM (hardware assisted virtualization)
Ahmed M. Azab, Peng Ning, Emre C. Sezer, and Xiaolan Zhang, "HIMA: A Hypervisor
Based Integrity Measurement Agent," in Proceedings of ACSAC 2009, December 2009.
Computer Science
June 29, 2012
ACNS 2012
9
VM Image Security Services
Nüwa—offline patching [ACSAC ’10]
Offline virus scanning [CCSW ’09]
Computer Science
June 29, 2012
ACNS 2012
10
Example Service: Nüwa（女娲）
• Nüwa – Offline Patching of VM Images
• Motivation
– Dormant VM images usually contain vulnerabilities
– Offline patching service
• A desirable security service in compute clouds
• Technical challenge
– Current patching system: Designed for running systems
– Pre- and post-processing scripts
– Examples: Stop/start daemons; Conditional updates
Computer Science
June 29, 2012
ACNS 2012
11
Nüwa Approach
• Two phases
– Phase 1: Automatic script rewriting
– Phase 2 (Leftovers): Resort to online updates
– Our research focus is on script rewriting
• Variations
– Standalone Nüwa: Offline patching of individual VM
images in emulated environments
– Mirage-based Nüwa: Batched offline patching using the
Mirage VM image library
Wu Zhou, Peng Ning, Xiaolan Zhang, Glenn Ammons, Vasanth Bala, Ruowen Wang, "Always
Up-to-date -- Scalable Offline Patching of VM Images in a Compute Cloud," in Proceedings of
ACSAC 2010, December 2010.
Computer Science
June 29, 2012
ACNS 2012
12
Some Evaluation Results
• Standalone Nüwa
– Base VM image
• 64-bit Ubuntu 8.04; 406 patches (collected on 10/26/2009)
• 402 patches can be applied offline (99%)
– Failure cases
• Mono-gac package and three others that depend on it
• Mirage-based Nüwa
– 100 VM images based on 32-bit Ubuntu 8.04
• Using 100 randomly selected subsets of basic Ubuntu tasks
– Top 8 security updates from Ubuntu Security Notices
• Ranked by Ubuntu popularity contest
• All data collected on 01/18/2010
Computer Science
June 29, 2012
ACNS 2012
13
Some Evaluation Results (Cont’d)
• Performance gain by standalone Nüwa
– About 4 times speedup
Offline
Online
120
Time (in seconds)
101.9
97.3
100
89.0
88.2
80
75.4
75.1
72.6
70.2
80.2
60
40
26.2
20
16.5
18.6
14.9
17.8
26.8
16.6
23.1
21.3
0
bash
dbus
libfreetype6
libpng12-0
libxml2
perl
sambacommon
udev
Average
* “Average” refers to the average of all 402 packages.
Computer Science
June 29, 2012
ACNS 2012
14
Some Evaluation Results (Cont’d)
• Additional speedup by Mirage-based Nüwa
– Another 2 – 10 times
Speedup of Mirage-based batch patching
over one-by-one patching
12
dbus
10
dash
libdbus
Speedup
8
libfreetype6
6
Libpng12-0
4
libglib2.0-0
libxml2
2
udev
0
10
20
30
40
50
60
70
80
90
100
Number of images patched
Computer Science
June 29, 2012
ACNS 2012
15
Hypervisor Integrity Services
HyperSentry
[CCS ’10]
Computer Science
June 29, 2012
ACNS 2012
16
Example Service: HyperSentry
• Why HyperSentry?
– Hypervisor is the highest privileged software
– Compromise of hypervisor  compromise of the system
– Hypervisors cannot be blindly trusted
• Example #1: Xen owning trilogy [BlackHat 2008]
• Example #2: VM Ware ESX 3.x
– 6/18/12: 67 Secunia advisories; 562 vulnerabilities; 7%
Secunia advisories not patched
– Hypervisor's code base is growing  More vulnerabilities?
Computer Science
June 29, 2012
ACNS 2012
17
HyperSentry
• HyperSentry
– Stealthy and in-context measurement of hypervisor integrity
• Challenges
– A fundamental problem
• How to measure the integrity of the highest privileged software?
– Hypervisor has full control of the software system (most of
the time)
• Scrubbing attacks
• Tampering with the measurement agent
• Tampering with the measurement results
– Relying on a higher privileged software goes back to the same
problem
Computer Science
June 29, 2012
ACNS 2012
18
The HyperSentry Approach
• HyperSentry
– A generic framework to stealthily measure the integrity of a
hypervisor in its context
• Key ideas
– Allow the measurement software to gain the highest
privilege temporarily
– Measurement is triggered stealthily
• Scrubbing attacks
– Isolate measurement results from the hypervisor
Ahmed M. Azab, Peng Ning, Zhi Wang, Xuxian Jiang, Xiaolan Zhang, Nathan C.
Skalsky, "HyperSentry: Enabling Stealthy In-context Measurement of Hypervisor
Integrity,” in ACM CCS 2010, October 2010.
Computer Science
June 29, 2012
ACNS 2012
19
Case Study: Verifying the Integrity of Xen
• Integrity measurement
– Code: SHA-1 hash of Xen's code; Control flow verification
– Date: Detect unauthorized sharing of physical pages across guest VMs
• Performance (on IBM HS21 XM blade server)
– End-to-end execution time: 35ms
– Periodical measurement: Every 8s: 2.4%; every 16s: 1.3%
Computer Science
June 29, 2012
ACNS 2012
20
Isolated Execution Bypassing Hypervisor Control
SICE
[CCS ’11]
Computer Science
June 29, 2012
ACNS 2012
21
SICE: Strongly Isolated Computing Environment
VM
VM
VM
VM
VM
Legacy Host
Legacy Host
(Hypervisor/OS)
(Hypervisor/OS)
Hardware
VM/
Workload
SICE
Hardware
Virtualized Platform
Virtualized Platform with SICE
Ahmed Azab, Peng Ning, Xiaolan Zhang, “SICE: A Hardware-Level Strongly
Isolated Computing Environment for x86 Multi-core Platforms,” in Proceedings
of ACM CCS 2011, October 2011.
Computer Science
June 29, 2012
ACNS 2012
22
Foundation of SICE
• System Management Mode (SMM)
– x86 operating mode for system management functions
– Single entry point: System Management Interrupt (SMI)
– SMRAM: Isolated from the rest of system
• Not accessible by the system software (e.g., hypervisor)
– AMD processors implementation
• Resizing the SMRAM at runtime
• Separate SMRAM range for each CPU core
– Main challenges
• SMM has escalated privileges
• The CPU runs slowly and has limited functionality
Computer Science
June 29, 2012
ACNS 2012
23
Foundation of SICE (Cont’d)
• Trusted/Secure boot
– Building the trust chain during system initialization
• C-RTM  BIOS  Boot loader  Initial hypervisor/OS image
– Secure hardware extensions (e.g., the TPM)
• Seals and authenticates the measurement output
– Main challenge
• Trust cannot be sustained due to potential runtime attacks
Computer Science
June 29, 2012
ACNS 2012
24
SICE Architecture
Special hypervisor:
•• Confines
Implementation
requirements
the isolated workload
• Trusted by the host
Provided by the user
• Secure boot
1. Hardware: AMD processors
• Consists of 300 SLOC to:
1. Maintain the isolation
2. BIOS: Load the SMI handler
2. Initialize and attest to the
Communication
channel:
3. Legacy host: Provide the communication
channel
isolated
environment
•Providing hardware services
VM
VM
Isolated Workload
(VM)
APP
Security Manager
(hypervisor)
SMI
Handler
(SICE)
Isolated Environment
SMRAM
Hypervisor/OS
Legacy Host
Hardware
DMA Devices
Trusted
Untrusted
Isolated
CPU, TPM, BIOS, etc…
Virtualized Platform with SICE
Computer Science
June 29, 2012
ACNS 2012
25
SICE Operations
• Operating modes
– Time-sharing Mode (an intermediate step)
– Multi-core Mode
• Remote attestation
– The initial image of the isolated environment
– Secure communication with remote users
Computer Science
June 29, 2012
ACNS 2012
26
Time-sharing Mode
• Initialization
– Secure boot
– SMI to load the isolated workload
• Execution environment switching
– SMI to trigger the isolated environment
– Changing the saved CPU state
– Changing the SMRAM memory range
– Fresh CPU start in the new environment
Workload
Image
Running
Workload
SMI
Security SMI Handler
SMI
Manager
(SICE)
(Hypervisor)
Image
SMRAMSMRAM
Legacy Host
Isolated Env.
Hardware
BIOS/TPM
– SMI to return to the legacy host
• Termination
Computer Science
June 29, 2012
ACNS 2012
27
Multi-core Mode
• Concurrent sharing of the hardware
– Good utilization
– One or more CPU cores are assigned to either
• The isolated environments
• The legacy host
• Main challenges
– Event isolation
– Memory isolation
Computer Science
June 29, 2012
ACNS 2012
28
Multi-core Mode (Cont’d)
• General multi-core processor architecture
AMD processors:
• Define the SMRAM
• SMM_BASE
• SMM_MASK
Core n
Core 1
Core 0:
• Registers
• MSRs
• L-APIC
• L1 Caches
Memory Control Hub (North Bridge):
• Configuration registers
• L2 Cache
• I/O registers
Computer Science
Core n
Core 1
Core 0:
• Registers
• MSRs
• L-APIC
• L1 Caches
Memory Control Hub (North Bridge):
• Configuration registers
• L2 Cache
• I/O registers
June 29, 2012
ACNS 2012
29
Multi-core Mode Operations
• Initialization
– Secure boot
– Loading the isolated workload
Workload
Image
SMI
Handler
SMI Security
Manager
(SICE)
SMRAM
Legacy Host
Hardware
Computer Science
June 29,
2012
ACNS 2012
30
Multi-core Mode Operations
• Running the isolated environment: The isolated core
– Changing saved CPU state
• E.g., page tables, interrupt descriptor, instruction & stack pointers
– Changing the SMRAM memory range (password stored in SMRAM)
– Fresh CPU start in the isolated environment
Workload
Isolated
Image
Workload
Security
Manager
Legacy Host
Running Isolated
Workload
Image
Workload (VM)
SMI
Handler
(SICE)
Security
Security Manager
Manager
(Hypervisor)
Isolated Env.
Legacy Host
SMRAM
SMRAMSMRAM
Isolated CPU Core
Host CPU Core
Hardware
Computer Science
SMI
Handler
(SICE)
June 29,
2012
BIOS/TPM
ACNS 2012
31
Multi-core Mode Operations
• Running the isolated environment: The host core
– Return to the legacy host
• No environment switching necessary
Workload
Isolated
Image
Workload
Security
Manager
Legacy Host
Running Isolated
Workload (VM)
SMI
Handler
(SICE)
Security Manager
(Hypervisor)
Isolated Env.
Legacy Host
SMRAM
SMRAM
Isolated CPU Core
Host CPU Core
Hardware
Computer Science
SMI
Handler
(SICE)
June 29,
2012
BIOS/TPM
ACNS 2012
32
Multi-core Mode Event Isolation
• Event isolation
– Prevent the legacy host and the isolated workloads send
events to each other
– Events between cores: Inter-Processor Interrupts (IPI)
• Two types of IPIs
– Maskable IPIs
• Can be blocked by recipient core’s APIC
– Non-maskable IPIs
• Can be controlled by Global Interrupt Flag (GIF)
• Clear GIF to ignore or hold all IPIs
Computer Science
June 29, 2012
ACNS 2012
33
Multi-core Mode Event Isolation (Cont’d)
• Protecting the host core
– The security manager runs as a thin hypervisor
• Prevents the isolated workload from privileged hardware access
• Protecting the isolated core
– The security manager
• Clear GIF
• All IPIs are disabled
– The isolated workload
• Set the GIF and re-enable maskable IPIs
• IPIs will cause a VM exit, which are examined by the security
manager for processing
Computer Science
June 29, 2012
ACNS 2012
34
Multi-core Mode Memory Isolation
• Memory Isolation
– Assigning different memory ranges to different CPU cores
• Memory double-view technique
• Each CPU core has its own SMRAM
Set by the SMI
handler
Shared
Memory
Set by the security
manager
Isolated
Workload
Registers:
•cr3
MSRs:
•SMM_Mask
•SMM_Address
Host Core(s)
Security
Manager
SMI Handler
Physical Memory
Computer Science
June 29, 2012
Registers:
•Nested cr3
MSRs:
•SMM_Mask
•SMM_Address
Isolated Core(s)
ACNS 2012
35
The SICE TCB
• The isolated environment
– Hardware + BIOS + SMI handler (300 SLOC)
• The legacy host
– Hardware + BIOS + SMI handler + The security manager
– Similar to micro-hypervisor approaches
Computer Science
June 29, 2012
ACNS 2012
36
SICE Prototype & Evaluation
• IBM LS 22 blade server
• Modifying Qemu/KVM to support a SICE isolated Linux guest
– No disk emulation: RAM disk image
• Time needed for context switching: 46 µs
• Evaluation results
Computer Science
June 29, 2012
ACNS 2012
37
Conclusion
• Infrastructure security of cloud computing
– Necessary for new research
– Better security protection for cloud workloads
• Security architecture for cloud computing
–
–
–
–
–
Hypervisor-based runtime security services
VM image security services
Hypervisor integrity services
Isolated execution bypassing hypervisor control
Not necessarily complete
• Hopefully a guidance/framework for innovative ideas
• Stay relevant!!!
Computer Science
June 29, 2012
ACNS 2012
38
Questions?
Thank You!
Computer Science
June 29, 2012
ACNS 2012
39