Secure In-VM Monitoring Using
Hardware Virtualization
Monirul Sharif, Wenke Lee,
Weidong Cui, and Andrea Lanzi
Presented by Tyler Bletsch
Introduction
• Problem: Kernel level rootkits can kill security
monitoring software
• Solution: Virtualization allows the monitor to
live outside the VM
• Problem: Out-of-VM monitoring can be very
expensive
– Why?
2
Out-of-VM monitoring
Inspection
DP
CP
K
DK
A
Program data
Program code
Hook
Hook data
Adversary program
H
CM
DM
R
Handler
Monitor code
Monitor data
Response
3
Problem with out-of-VM monitoring
• Significant performance impact
– VM context switch is expensive
– Introspection is hypercall-driven: expensive
• We want a monitor without these problems
• Performance guarantees:
– (P1) Fast invocation:
No privilege change for handlers
– (P2) Data read/write at native speed:
No penalty to access DM or DP
4
Solution: Secure In-VM Monitoring
DP
CP
K
DK
A
Program data
Program code
Hook
Hook data
Adversary program
H
CM
DM
R
Handler
Monitor code
Monitor data
Response
5
Security guarantees
• Run the monitor inside the VM context,
but provide out-of-VM-style security guarantees:
– (S1) Isolation of monitor code CM and data DM:
Adversary can’t access the monitor at all
– (S2) Designated point for switching into CM:
There’s only point entry point (tiny attack surface)
– (S3) The handler is called iff the hook is triggered:
The hook calls the handler; nobody else can
– (S4) Behavior of M is not maliciously alterable:
No dependency on monitored data or code
6
How to achieve this?
• Separate address space with a
one-way page table for the monitor
• Small entry/exit gates into this address space
– Invocation checker verifies that the call is legit
7
The SIM address space
SIM Data/Code
• The monitor itself
• Visible only within SIM address space
Invocation checker
• Verifies call chain is legit
• Visible only in SIM space
Entry/exit gates
• Visible in both
• Writable only in SIM space
• Tiny, well crafted
Kernel code/data
• Not executable in SIM space
(can't accidentally run insecure code)
8
Changing the address space
• CR3 register indicates page table directory
– Hypervisor controlled
• To avoid hypercall, use Intel CPU's
"CR3_TARGET_LIST" to freely switch between:
– P_SHADOW: Process's normal space
– SIM_SHADOW: Restricted SIM space
9
Entry/exit gates
• Only point of entry for SIM address space
• Entry:
–
–
–
–
–
–
Disable interrupts
Save CPU state to the stack
Switch address space
Re-disable interrupts
Switch stack to a SIM-restricted one
Run invocation checker
• Exit:
– Restore stack, page table, CPU state
– Re-enable interrupts
– Jump to return point (usually the point of origin)
10
Invocation checker
• Obviously can't rely on stack
• Use CPU debug registers to track jump history
• Lock these registers from modification by VM
11
Security monitor
• Can't rely on untrusted code
– No kernel functions, dynamic libraries, etc.
• Need to traverse kernel data structures
just like the out-of-VM technique (VMI)
• Hardware access granted by hypervisor API
12
Performance requirements met?

(P1) Fast invocation:
Yes: No privilege change needed;
just a page table switch

(P2) Data read/write at native speed:
Yes: Pages are mapped, accessed natively
13
Security requirements met?

(S1) Isolation of monitor code CM and data DM:
Yes: Monitor memory isn't mapped to the process space

(S2) Designated point for switching into CM:
Yes: Entry/exit points are the only way in

(S3) The handler is called iff the hook is triggered:
Yes: Hook calls handler; calls pass the invocation checker

(S4) Behavior of M is not maliciously alterable:
Yes: Untrusted code is non-executable in SIM space
14
Ah, but what if they...
• Jump into the entry gate after the interrupt
disable instruction (CLI)?
– It gets re-disabled again after the page table switch
• Switch to SIM_SHADOW directly?
– The next instruction is no longer executable
– But what if they do it right before a SIM page?
•Any page after an unallocated address is made
non-executable
15
Implementation
• Linux KVM host, Windows XP guest
• To initialize from a trusted boot state, a driver:
– Reserves virtual address space for entry/exit gates
– Has the hypervisor construct a SIM page table
– Loads the security monitor into kernel space and
has the hypervisor remap it into SIM space
– Informs the hypervisor of each hook and handler;
hypervisor generates entry/exit routines accordingly
• Memory protection:
– Don't check every time CR3 is changed,
check new page table entries as they are created
16
Evaluation: Monitor invocation
• Time to enter and exit a no-op handler
– SIM:
– Out-of-VM:
0.469 μs
5.055 μs
• Factor of 10.7 improvement
17
Evaluation: Process creation monitor
• Hook process creation
• Traverse kernel data to find process name
• Process must be on whitelist to proceed
– If not, skip and return failure
• Average time for process creation:
– Traditional In-VM approach: 3.487 μs
– SIM approach:
3.967 μs (+13.7%)
– Out-of-VM approach:
28.039 μs (+690.5%)
18
Evaluation: System call tracing
• Make note of every system call,
emit trace every 10,000 calls
• Macrobenchmarks from PCMark 05 suite
Benchmark
Memory Latency
Bare
Out-of-VM
SIM
overhead overhead
10.42 MAcc/s
84.58%
7.97%
HTML Render
1.12 pg/s
52.42%
5.83%
File Compress
3.4 MB/s
3.97%
0.59%
File Encrypt
20.56 MB/s
7.85%
0.89%
File Decrypt
78.21 MB/s
2.53%
0.45%
HDD
15.29 MB/s
41.68%
3.74%
82.73 pg/s
128.84%
9.64%
46.10%
4.15%
Text Edit
Average
-
19
Conclusion
• SIM: The best of both worlds
– Security of out-of-VM solution
– Performance of traditional in-VM monitoring
20
Concerns
• Very difficult to develop monitors
– No libraries
– No system calls
– Porting a full-fledge anti-malware package?
• Interrupt race condition in entry code?
Normal entry
Attacker entry
CLI
PUSHA
MOV EAX, SIM_SHADOW
MOV CR3, EAX
Address
space
switch
Interrupt now?
CLI
MOV [P_ESP] ESP
MOV ESP, [SIM_ESP]
JMP INVOKATION_CHECK
21
Discussion
22