Computer Science
HyperSentry: Enabling Stealthy In-context
Measurement of Hypervisor Integrity
Ahmed M. Azab, Peng Ning, Zhi Wang, Xuxian Jiang
North Carolina State University
Xiaolan Zhang
Nathan C. Skalsky
IBM T.J. Watson Research Center
IBM Systems & Technology Group
1
Background
• Hypervisors are critical to virtualized platforms
• Can hypervisors be blindly trusted?
– Two backdoors in Xen [BlackHat 2008]
– Xen 3.x: 10 Secunia advisories; 17 vulnerabilities.
– VM Ware ESX 3.x: 49 Secunia advisories; 362 vulnerabilities.
– Existing hypervisor's code base is growing
• Need mechanisms to ensure hypervisor integrity
– HyperSentry: Measure the hypervisor integrity at runtime
Computer Science
2
Challenges
• A fundamental problem
– How to measure the integrity of the highest privileged
software?
• Hypervisor has full control of the software system
– Scrubbing attacks
– Tampering with the measurement agent
– Tampering with the measurement results
• Relying on a higher privileged software goes back to
the same problem
Computer Science
3
The HyperSentry Approach
• HyperSentry
– A generic framework to stealthily measure the integrity of a
hypervisor in its context
• Key ideas
– Allow the measurement software to gain the highest
privilege temporarily
– Measurement is triggered stealthily
• Scrubbing attacks
– Isolate measurement results from the hypervisor
Computer Science
4
Foundation of HyperSentry
• System Management Mode (SMM)
– x86 operating mode for system management functions
– SMRAM can be locked to prevent all access to it except
from within the SMM
• Hypervisor cannot access the SMRAM once locked
– System Management Interrupt (SMI) only handled by SMI
handler in SMRAM
• SMI bypasses hypervisor’s control
– Provides the isolation required for HyperSentry
– Main challenges
• How to retrieve the needed context for hypervisor?
• How to attest to the measurement output?
Computer Science
5
Foundation of HyperSentry (Cont’d)
• Out-of-band communication channel
– Triggers a System Management Interrupt (SMI)
– Out of the control of the hypervisor
– Example: IPMI
• Uses a microcontroller on the motherboard
• Hard-wired to GPI chip to trigger SMI
• Not under the control of the Hypervisor
– Main challenge
• How to prevent or detect hypervisor’s intervention (e.g., reprogram
APIC)?
Computer Science
6
HyperSentry Architecture
VM
VM
VM
Guest (non-root) Mode
Host (root) Mode
Hypervisor
Measurement
Agent
Hardware
Virtualized Platform
Computer Science
SMI
Handler
Trusted
Components
are Shaded
in Green
System
Management
Mode
BMC/IMM
Remote
Verifier
7
In-context Integrity Measurement
• Challenges
– How to detect the intercepted CPU operation mode?
• Hypervisor or guest VM?
– How to retrieve the context needed for measurement?
• E.g., CR3 and page table
• Solution
– Inject a privileged instruction to force the CPU to fall back
to the hypervisor mode
– Run the measurement agent in the same context as the
hypervisor
• Agent runs in a protected execution environment
Computer Science
8
In-context Integrity Measurement
Guest VM
Execution Path
SMI
RSM
Prepare SMM fallback
Inject privileged
instruction and flush cache
Privileged instruction
Guest (non-root) Mode
Hypervisor
Host (root) Mode
SMI
RSM
VM exit handler
SMI
The measurement
agent
Hardware
Verify the measurement
agent
Store measurement output
System Management Mode
PC (cache misses = 0)
1)
Computer Science
APIC (SMI on PC overflow)
9
Stealthy Invocation
• Is out-of-band invocation sufficient to achieve
stealthy invocation?
– Unfortunately …
Computer Science
10
A Variation of Scrubbing Attack
VM
VM
VM
Guest (non-root) Mode
Host (root) Mode
SMI
Handler
Typical
Scenario
System
Management
Mode
Hypervisor
Hardware
BMC/IMM
Remote
Verifier
Computer Science
11
A Variation of Scrubbing Attack
VM
VM
VM
Guest (non-root) Mode
Host (root) Mode
SMI
Handler
Attack
Scenario
System
Management
Mode
Hypervisor
Hardware
BMC/IMM
Compromised hypervisor cannot intercept SMIs. But what if it
tries to block real SMIs and generate fake ones?
Computer Science
Remote
Verifier
12
Thwarting this Scrubbing Attack
• Can we prevent the hypervisor from blocking SMIs?
– Not possible with existing hardware
• Solution
– Detecting fake SMIs generated by the (compromised)
hypervisor
• Verifying status registers to ensure that the measurement is invoked
by the out-of-band channel
– Key reason: HW SMI and SW SMI are distinguishable
Computer Science
13
Stealthy Invocation
CPU
Core 0
CPU
Core n
CPU
Core 1
SMI
- All status register are non writable
- Measurement is invoked only if all
other bits are 0
- A fake SMI is easily detectable
Memory Control Hub (North Bridge)
SMI_EN
1
GPI_ROUT
0 1
ALT_GPI_SMI_EN
1
10 9
SMI_STS
0
0 …..0 1 0…….0
GPI 0 BMC
ALT_GPI_SMI_STS 0 ……………….0 1
IO Control Hub (South Bridge)
IPMI AMM
SSH
Remote
Verifier
Target Platform (IBM HS21XM Blade Server)
Computer Science
14
Attesting to the Measurement Output
• Challenge
– Absence of a dedicated hardware for attestation
• The hypervisor controls the hardware most of time
• Solution
– Providing the SMRAM with a private key
– Using this key to attest to the measurement results
Computer Science
15
Attesting to the Measurement Output
Ksmm-1{ Output|Nonce}
Remote
Verifier
Guest
Bootstrapping
VM
SMI handler
Attestation
request
SMM private key
Guest Mode
Hypervisor
Host Mode
Initialization
code
KAIK-1{Ksmm |Handler|Nonce}
System Management
Mode
Hardware
Computer Science
Integrity
Ksmmmeasurement
output
Ksmm
-1
TPM
SMM
public key
16
Security Analysis
• Stealthy Invocation
– If configurations are not changed  guaranteed by hardware
– If configurations change  fake SMIs are detectable
• Verifiable Behavior
– The measurement agent is measured every time before it executes
• Deterministic Execution
– The measurement agent possesses full control over the system
• In-context privileged measurement
– Guarantee falling back to the hypervisor mode
– The measurement agent runs in the same context as the hypervisor
• Attestable output
– The measurement output is signed by a verifiable and protected key
Computer Science
17
HyperSentry Evaluation
• IBM HS21XM blade server
• Measuring the Xen hypervisor
– End-to-end execution time: 35 ms
– Periodical measurement:
• Every 8 seconds: 2.4% overhead; every 16 seconds: 1.3% overhead
Computer Science
18
Conclusion
• HyperSentry
– A novel framework for measuring the integrity of the most
privileged system software
– A measurement agent for the Xen hypervisor
– Low overhead
• Next step
– Measurement agent for Linux/KVM
– Verifying the hypervisor’s dynamic integrity
Computer Science
19
Questions?
amazab@ncsu.edu
Computer Science
20