Stealthy Malware Detection Through VMM-Based
“Out-of-the-Box” Semantic View Reconstruction
November 2007
ACM: Association for Computing Machinery
Authors: Xuxian Jiang-North Carolina State University
Xinyuan Wang-George Mason Univeristy &
Dongyan Xu-Purdue University

Semantic : of, pertaining to, or arising from the different meanings of words or other symbols
Semantics : the study of meanings: the language used to achieve a desired effect on an audience especially through the use of words with novel or dual meanings


There is a recent trend in malware to equip the software with stealthy techniques to detect, evade and avoid malware detection attempts The fundamental limitation of current hostbased anti malware systems is they run inside the host they are protecting. This is called "in-the-box" which makes them vulnerable to counter detection and avoidance by certain malwares.

To fix this limitation, many solutions are using Virtual Machine technologies and placing the malware detection facilities outside of the protected VM bubble. This is called "out-of-thebox". Yet, they gain breaking into to at the cost of loosing the internal semantic view of the host which is enjoyed by the "inthe-box" approach. This causes a technical challenge called the
"semantic gap".


The paper about the design, implementation and evaluation of VM Watcher and "out of the box" approach that overcomes the semantic gap challenge.

New technique called "guest view casting"

Developed to reconstruct internal semantic views (files, ps and kernel modules) of VM from the outside, rather than typical inside approach.


New technique casts semantic definitions of guest OS Data Structures and functions

Puts on the Virtual Machine Monitor (VMM)
Level VM state

Semantic view reconstructed from multiple perspectives

Reconstruct these details for system call events
(ps, call #, parameters, & return value) in the
VM & increases the semantic view.


With semantic gap bridged we identify two unique malware detection capabilities:


View comparison-based malware detection: and it's demonstration in rootkit detection
Out of the box deployment of host based anti malware software with improved detection accuracy & tamper resistance


Internet malware-rootkits and bots are getting very sneaky and elusive. They hide their presence from detection factilities & anti malware software

Host based anti malwared systems are installed and executed inside the hosts they are monitoring and protecting: “in the box”

This makes the anti malware system visible, tangible, and unavoidable to the malware inside the host



Now with Virtual Machine technologies we can use this to our advantage. Use the strong isolation and confines ps inside VM so that even if it's compromised by malware, it will be hard to compromise systems outside the VM
“semantic gap” between the VM view from inside the box vs outside the box

Inside views: ps, files, kernel modules

Outside views: memory pgs, registers, disk blocks


Advantages to both views.

VM Watchera VMM based “out of the box” approach overcomes the semantic gap challenge


It starts the Virtual Machine view in a non intrusive manner so it can inspect low level VM states without influencing the VM's execution
“guest view casting” a new technique


This new approach reconstructs the VMs internal view: files, dir, ps, and kernel level modules for “out of the box” malware detection

Based on the observation that the guest
Operating System of a VM provides all the necessary definitions of guest data structures & functions to construct the VM sematic view & cast them on the VMM level observation

Also externally remake the sematic view of the target Virtual Machine


VM Watcher should not disturb the system state of the VM being monitored

VM Watcher should narrow the sematic gap so that malware detection systems run inside the
VM can also run outside the VM

VM Watcher should be generic and applicable to a wide range of existing VMMs.
 2 approaches: full virtualization (VMWare,
QEMU) & para virtualization (Xen, User Mode
Linux)


Non Intrusive VM Introspection: provide low level VM states externally. Non intrusive technique to gain full VM state including registers, memory & disk

Guest View Casting: external reconstruction of the sematic level view of VM thus bridging the semantic gap


VM Watcher w/ 4 existing VM's: VMWare,
QEMU, Xen & UML. The implemenation details:

Open source VMM: QEMU, Xen & UML. Close source: VMWare only exposes raw disk blocks & raw memory pgs. Open source allows full access to low level VM states and events


3 unique detection & monitoring capabilities:
 (i) view comparison: based malware detection and its demonstration in rootkit detection
 (ii) “out-of-the-box” deployment of off-theshelf anti malware software with improved detection accuracy and tamper-resistance
 (iii) nonintrusive: system call monitoring for malware and intrusion behavior observation

 Evaluation experiments with real-world malware
 Includes elusive kernel-level rootkits
 Demonstrates VMwatcher's practicality and effectiveness
 #1: Viewed comparison on volatile states
 #2: Viewed comparison on persistent states
 #3: Viewed comparison on both volatile & persistent states
 #4: Cross platform malware detection


Involves Windows kernel FU rootkit. It runs and hides in process w/ PID 336. VMWare running w/ host OS is Scientific Linux 4.4 & guest OS is
Windows XP SP2.

Windows cmd shell PID 1080 is c reated and invokes the FU rootkit to hide ps 336. The hidden ps is running SSH. The Windows Task
Mgr does not list the SSH client ps indicating that this ps has been hidden

Exposed by VM Watcher external view.


The small box w/ solid lines indicates the
SSHClient.exe ps which is not shown by Win
Task Mgr.

VM Watcher can be readily adopted by real world honey pots to detect in the wild rootkit attacks.

Also recent incidents show the same FU rootkit has been actively used to hide the presence of advanced bots


Advanced Linux kernel rootkit that replaces kernel level function pointers to hide files & ps

Adore-ng infection on a Xen Fedora Core 4 Virtual
Machine

Four xterm windows

0: inside the VM where adore-ng kernel mod is loaded w/ backdoor ps PID 1490

1: external view of VM: mounted devices

2: files under the dir /root/demo in the VM

3: current running ps inside VM

Software VMM Guest OS
Symmanec Anti Virus 10.1.0396
Windows Defender/Malicious
Software Removal Tool
Trend Micros Server Protect for Linux
2.5
Kaspersky Anti-Virus 5.5
F-Secure Anti—Virus 5.20
VMWare Server 1.0.1
VMWare Server 1.0.1/Xen 3.0.2-2
VMWare Server 1.0.1/Xen 3.0.2-2
VMWare Server 1.0.1/Xen 3.0.2-2
VMWare Server 1.0.1/Xen 3.0.2-2
Windows XP/Red Hat 7.2
Windows XP/Red Hat 7.2
Red Hat FC4/Windows XP (SP2)/Red
Hat 7.2, 8.0, 9.0
Red Hat FC4/Windows XP (SP2)/Red
Hat 7.2, 8.0, 9.0
Red Hat FC4/Windows XP (SP2)/Red
Hat 7.2, 8.0, 9.0
Frisk F-PROT Antivirus for Linux
McAfee Virus Scan 4.24.0
Sophos Anti Virus 4.05.0
Tripwire 4.05.0 (Open Source)
Xen 3.0.2-2/QEMU 0.8.2
UML 2.4.24
QEMU 0.8.2
UML 2.4.24
Red Hat 7.2, 8.0, 9.00
Red Hat 7.2, 8.0, 9.00
Red Hat 7.2, 8.0, 9.00
Red Hat 7.2, 8.0, 9.00
ClamAV 0.88.5 (Open Source) UML 2.4.24
Red Hat 7.2, 8.0, 9.00
Host OS
Windows XP(SP2)
Windows XP(SP2)
Scientific Linux 4.4
Scientific Linux 4.4
Scientific Linux 4.4
Scientific Linux 4.4
Red Hat
Red Hat
Red Hat
Red Hat


VM Watcher is a VMM approach that enables out of the box malware detection

Addresses the semantic gap challenge

VM Watcher has stronger tamper resistance by moving anti malware facilities out of the monitored VM while maintaining a current semantic view of the VM “inside the box” via external semantic view reconstruction


VM Watcher prototype on Linux and Windows platforms shows it's practical nature and effectivness

The experiments with real world self hiding rootkits demonstrates the power of new malware detection capabilites introducted by
VM Watcher


Good points: very concrete experiments shown towards end of the paper that brought it all together

Used a variety of open source & proprietary
Operating Systems and current anti virus softwares in experimentations

Bad points:Was not able to discuss
Experiments 2 and 4 due to time constraints
(me)

Guest view casting Figures were confusing


Vocabulary used was very extensive and advanced

With the technical nature of the paper, the vocabulary used should have been more basic in nature to facilitate better understanding

Had to reread the paper a few times to understand the jist of the paper


Great experiments were done in relation to malware/rootkit detection

Virtual Machine experimentation was great.
Liked the use of open source VM's such as
Xen, QEMU, and UML.

Talked about different VM states: full vs para virtualization. Future work with this would be great.

Further discussion of honey pots and “in the wild” rootkit attacks would improve the paper