SPLIT PERSONALITY MALWARE
DETECTION AND DEFEATING IN
POPULAR VIRTUAL MACHINES
Alwyn Roshan Pais
Alwyn.pais@gmail.com
Department of Computer Science & Engineering
National Institute of Technology, Karnataka
1
2
OBJECTIVE
To study the VM detection techniques in popular
Virtual machines.
 Develop strategy to counter the detection.
 Prevent analysis aware malwares from detecting
VM.

3
PLAN OF ACTION
Introduction
 VM detection techniques
 Detection techniques in VMware, VirtualBox and
VirtualPC.
 Related Work
 Prevent Analysis aware malwares from detecting
VM.
 VMDetectGuard – Tool to mask VM detection :
Windows
 Optimization of VMDetectGuard
 Results

4
INTRODUCTION
5
MALWARE
 Malware: It is a collective term for any
malicious software which enters the
system without the authorization of the
user of the system.
 Anti-virus/anti-malware products do not
guarantee complete protection.
6
PRESENT SCENARIO
 Security
researchers use malware
analysis tools to build defenses against
the unknown malware forms.
 They then build patches for the newly
discovered vulnerabilities and exploits.
 Virtualization has emerged as a very
promising technology.
 Malware
analyst use Virtual Machine
Environment (VME), debuggers and
sandboxes in their analysis work.
7
VIRTUALIZATION
A
software based representation of a
computer that executes programs in the
same way as a real computer.
 Examples, VMware, Virtual PC,
VirtualBox.
 Advantages
Reduced capital and operational costs through
more efficient use of hardware resources.
 Simplifies maintenance .
 Improves scalability and deployment agility.
 Improves reliability.

8
BENEFITS OF VIRTUALIZATION TO
SECURITY RESEARCHERS
Researchers can intrepidly execute potential
malware samples without having their systems
affected.
 If a malware destabilizes the OS, analyst just needs
to load in a fresh image on a VM.
 Reduces time and cost.
 Increases productivity.

9
ANALYSIS AWARENESS FUNCTIONALITY
Malware developers have added a new
functionality to malware.
 Detect the presence of analysis tools such as
VMs, debuggers and sandboxes.
 Hide their malicious behavior on detection.
 Analysis Aware / Split Personality malware.

10
RELATED WORK
Carpenter (Carpenter et al., 2007) proposes two
mitigation techniques.
 They aim at tricking the malware by

1.
2.
Changing the configuration settings of the .vmx file
present on the host system and,
Altering the magic value to break the guest-host
communication channel.
11
DRAWBACKS OF THE FIRST APPROACH
The configuration options break the
communication channel between guest and host
not just for the program trying to detect the VM,
but for all the programs.
 Moreover the authors claim that these are
undocumented features and that they are not
aware of any side effects.

12
RELATED WORK
The work by Guizani (Guizani et al., 2009)
provides an effective solution for Server-Side
Dynamic Code Analysis.
 Small part of the solution deals with tricking the
Split Personality malware that employ Memory
Detection and VM Communication Channel
Detection techniques.

13
RELATED WORK

Kalpa Vishnani et. al. 2011: Masks all the
detection techniques used in Vmware.
14
RELATED WORK

Other works concentrate
Detecting this category of malwares
 Running in host machine

Save the current state
 quickly restore to previous state


Virtual machines in the order of market share

VMware, Virtual PC, and Virtual Box.
15
VM DETECTION TECHNIQUES
Hardware fingerprinting
 Registry Check
 Process and File Check
 Memory Check
 Timing Analysis
 Communication Channel Check
 Invalid Instruction Check

16
HARDWARE FINGERPRINTING
Involves looking for specific virtualized
hardware.
 VMs give an abstracted view of many hardware
components.
 Querying for such components reveals VM
presence.
 For Example: BIOS, Motherboard, SCSI
Controllers, USB Controllers, etc.

17
HARDWARE FINGERPRINTING RESULTS
18
REGISTRY CHECK
 The
registry entries contain hundreds of
references to the string containing the name
of the VM, Ex. “Vmware”, VirtualPC and
VirtualBox.
 Checking the registry values for certain
keys clearly reveals the VM presence.
19
REGISTRY CHECK

For Example:
HKEY_LOCAL_MACHINE\HARDWARE\DEVICEMAP\Scsi\Scsi
Port1\Scsi Bus 0\Target Id 0\Logical Unit Id
0\Identifier
VMware, VMware Virtual S1.0
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class
\{4D36E968-E325-11CE-BFC108002BE10318}\0000\DriverDesc
 VMware SCSI Controller
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class
\{4D36E968-E325-11CE-BFC108002BE10318}\0000\ProviderName
 VMware, Inc.
20
PROCESS AND FILE CHECK

Check - VM specific processes and files presence

Eg.

VBoxService.exe : In VirtualBox for synchronization
with host

drivers like “vboxhook.dll” and “vpcbus” driver
present in %SYSDIR%/drivers
21
MEMORY CHECK
This involves looking for values of critical operating
system data structures.
 These data structures are relocated on a virtual
machine so that they do not conflict with the host
system's copies.
 Store Interrupt Descriptor Table (SIDT), Store
Local Descriptor Table( SLDT), Store Global
Descriptor Table (SGDT), Store Task Register
(STR), Store Machine Status Word (SMSW)
 Redpill.exe, ScoopyNG.exe use this method.

22
TIMING ANALYSIS
Obvious yet rare attack.
 Involves looking at a local Time Stamp Counter
(TSC) value.
 By noting down the time difference VM presence
is detected.

23
VM COMMUNICATION CHANNEL CHECK
This check involves detecting the presence of a
host-guest communication channel.
 IN instruction and a magic number ‘VMXh’
 VmDetect.exe uses this check.
 Not applicable to VirtualPC and VirtualBox.
 Runs in VMware without exception.

24
INVALID OPCODE CHECK



Specific to VirtualPC
Uses certain opcodes for guest host
communication
In host system raise exception and no exception
in VirtualPC.
25
VMWARE DETECTION
HARDWARE FINGERPRINTING

hardware details


Windows Management Instrumentation (WMI)
contains classes


motherboard serial number, graphics card and
network adapter captions
hardware, display, registry etc.
Check for VM specific strings
26
REGISTRY CHECK

Windows Registry stores
configuration settings
 low-level operating system components
 Applications running


Check for
Strings like “VirtualPC”, “VBOX”, “VirtualBox”
 value that is specific to the corresponding virtual
machine being testing on.

27
PROCESS AND FILE CHECK

Check - VM specific processes and files presence

Eg.

VBoxService.exe : In VirtualBox for synchronization
with host

drivers like “vboxhook.dll” and “vpcbus” driver
present in %SYSDIR%/drivers
28
MEMORY CHECK


involves looking at the values of specific memory
locations
STR (Store Task Register)

stores the selector segment of the TR register (Task
Register) in the specified operand (memory or other
general purpose register).

Value specific in Virtual Machine
29
INVALID OPCODE CHECK



Specific to VirtualPC
Uses certain opcodes for guest host
communication
In host system raise exception.
30
DETECTION OF VM RUNNING LINUX

Techniques: (tested on Vmware)

Hardware Fingerprinting

Dmesg check - prints the message buffer of the kernel

/proc file system check - interface to internal data
structures in the kernel.

Communication channel check
31
DMESG AND /PROC FILE SYSTEM CHECK


Dmesg - prints the message buffer of the kernel
Shows diagnostic message showing presence of
hardware during boot


contain strings like “VMware”,
/proc file system - an interface to internal data
structures in the kernel

Contains system dependent information
32
COMMUNICATION CHANNEL CHECK

IN instruction

Raises exception ““EXCEPTION PRIV
INSTRUCTION” in host

Runs in VMware without exception

initiates guest to host communication by calling the “IN”
instruction.
33
VMWAREDETECT
Is the proof of concept tool.
 It employs the various VM detection techniques
to detect the presence of VMware virtual
machine.






Memory Check
VM Communication Channel Check
Hardware Fingerprinting
Registry Check
Timing Analysis
34
VMWAREDETECT
35
VIRTUALMACHINEDETECT - VIRTUALPC

Check using all the methods
In VirtualPC
In Native Machine
American Megatrenda
L900781
Graphics Card
Virtual PC Integration Components S3
Trio32/64
NVDIA GeForce 310
Baseboard Manufacturer
Microsoft co-orporation
LENOVO
System Name
USB Controller
VIRTUALXP
USB Virtualisation Bus Driver
User-think
Intel® 5 Series /3400 …
Registry Check
SCSI: HARDWARE\\DEVICEMAP\\Scsi\\Scsi Port 0\\Scsi
Bus 0\\Target Id 0\\Logical Unit Id 0
Virtual HD
Hitachi HDS721050CLA362
Control class for usb :
SYSTEM\\ControlSet001\\Control\\Class\\{36FC9E60C465-11CF-8056-444553540000}\\0000
USB Virtualisation Bus Driver
Intel® 5 Series /3400 …
Control class for graphics:
SYSTEM\\ControlSet001\\Control\\Class\\{4D36E968E325-11CE-BFC1-08002BE10318}\\0000
Virtual PC Integration Components S3
Trio32/64
NVDIA GeForce 310
Controlset for cd/dvd drive:
SYSTEM\\CurrentControlSet\\Enum\\IDE
Disk Virtual_HD____1._1__
Registry not found
Invalid Opcode
File Check
Vpcubus Driver (Virtual USB Bus Driver)
Did not raise exception
Raised exception
Present
Not Present
Vpcgbus Driver (Virtual PC Guest Bus Driver)
Present
Not Present
Vpcuhub Driver (Virtual USB Hub Driver)
Present
Not Present
Hardware Fingerprinting
BIOS
36
VIRTUALMACHINEDETECT - VIRTUALBOX
Virtual Box running windows
Host Windows Machine
BIOS
0
L900781
Graphics Card
Virtual Box Graphics Adapter
NVDIA GeForce 310
N/W adapter
AMD PCNET Family PCI Ethernet Adapter
WAN Miniport(SSTP) …
Processor
Null
CPU1
USB Controller
Std Open HCD USB Host Controller
Intel® 5 Series /3400 …
Dsdt: : HARDWARE\\ACPI\\DSDT
VBOX__
Registry not present
Scsi P0 : HARDWARE\\DEVICEMAP\\Scsi\\Scsi Port 0\\Scsi Bus
VBOX HARDDISK
Hitachi HDS721050CLA362
VBOX CD-ROM
Null
Hardware Fingerprinting
Registry Check
0\\Target Id 0\\Logical Unit Id 0
Scsi P1: HARDWARE\\DEVICEMAP\\Scsi\\Scsi Port 1\\Scsi Bus
0\\Target Id 0\\Logical Unit Id 0
Vedio Bios Version:
Oracle VM VirtualBox Version 4.1.2 VGA Bios Version 70.18.3E.00.05
HARDWARE\\DESCRIPTION\\System\VideoBiosVersion
System Bios Version:
VBOX-1
LENOVO-133
28 0
40 00
HARDWARE\\DESCRIPTION\\System\SystemBiosVersion
Instruction Check
STR (store task register)
37
File Check
VBOXHook.exe
Present
Not Present
VBOXTray
Present
Not Present
VBOXService.exe
Present
Not Present
VIRTUAL MACHINE DETECT
In VB
38
REMOTE DETECTION


Scenario

There is access to the terminal of a system

need not be administrator access
WMIC ( Windows management instrumentation
command line) is used
39
MASKING DETECTION OF VM



Using PIN API provided by Pin tool.
Can get all the instructions, the arguments and
return value
Steps followed for masking
Get each call made by binary.
 Check if matches a predefined list of calls. E.g.

RegEnumValueA
 Str
 LoadLibraryA
 __emit

40
MASKING DETECTION OF VM

Provide false values if
VM specific values are read (matched from
predefined list)
 Eg.

Registry read returns the value “VBOX”
 Pin Tool gets the return value and modifies it in runtime.
 Registry read function returns modified value

41
MASKING DETECTION OF VM


Binary does not detect – manipulated value
received.
This currently supports
64 and 32 bit OS
 64 and 32 bit applications

42
MASKING DETECTION OF VM
Load Binary
Detect if the binary
is 64 or 32 bit.
Detect the Underlying VM
Display the detection and
give option to user to
change it.
Detect the OS as 64/32
bit.
VirtualBox
Virtual PC
Invalid Opcode Check
Masking
Register Check
masking
Instruction Check
Masking
Register Check
Masking
File Check
Masking
File Check
Masking
Execution of loaded
binary completed
Feedback
43
Save to db for
further analysis
OUR APPROACH
44
OUR APPROACH
STEP 1:
Maintain a list of all the hardware as well
as registry querying API calls. Also
maintain a list of all the VM specific
instructions such as SIDT, SLDT, SGDT,
STR, IN.
45
OUR APPROACH



Following is a partial list of API calls to be monitored.
Hardware Querying APIs
 SetupDiEnumDeviceInfo
 SetupDiGetDeviceInstanceId
 SetupDiGetDeviceRegistryProperty
Registry Querying APIs
 RegEnumKey
 RegEnumValue
 RegOpenKey
 RegQueryInfoKeyValue
 RegQueryMultipleValues
 RegQueryValue
46
OUR APPROACH
Step 2:
Perform dynamic binary instrumentation
of the sample under test in order to obtain
its low level information as well as to
intercept all the API calls made by it.
 We hook into the sample under test by
means of .dll injection.
 This is achieved using the pin framework.
47
OUR APPROACH
Step3:
Check to see if the sample under test
makes a call or executes any of the
monitored API calls or instructions
respectively. If a match is found, set the
OUTPUT to “Split Personality Malware
Detected”. Also, log the activity and
provide fake values to the sample so as to
make it feel that it is running on a host
system.
48
IMPLEMENTATION
 Designed,
implemented and tested
VMDetectGuard.
 Implemented in the framework provided
by the Pin tool released by Intel
Corporation.
 Pin is a tool for the instrumentation of
programs.
 We made use of its framework to intercept
the various API calls and low level
instructions executed by the sample under
test.
49
COUNTERING HARDWARE
FINGERPRINTING
Hardware emulation.
 APIs that query for BIOS, Motherboard, Processor,
Network Adapter.
 Ex. VM returns a value “none” for motherboard
serial number. VMDetectGuard returns a more
appropriate string such as
“.16LV3BS.CN70166983G1XF” instead.

50
COUNTERING REGISTRY CHECK
VMDetectGuard monitors registry querying APIs
such as the following:
 RegEnumKey
 RegEnumValue
 RegOpenKey
 RegQueryInfoKeyValue
 RegQueryMultipleValues
 RegQueryValue
 If the output contains the string "VMware", our tool
replaces this string with a more appropriate value
that would have been returned on a non virtual
51
system.

COUNTERING MEMORY CHECK
SIDT, SLDT, and SGDT and STR instructions
are monitored.
 The values of the target registers are then
changed appropriately with the values that
would have been obtained on a host OS.

52
COUNTERING MEMORY CHECK
53
COUNTERING VM COMMUNICATION
CHANNEL CHECK
Monitor execution of the IN instruction.
 We change the value of the magic number .
 This leads to generation of “EXCEPTION PRIV
INSTRUCTION” exception.

54
COUNTERING TIMING ANALYSIS
Instructions such as CPUID and RDTSC (Read
Time Stamp Counter) are monitored.
 The tool maintains a log of each type of
instruction executed.
 If the threshold value for a particular type of
instruction is exceeded, it logs this activity too.
 Sample is tricked by deleting the CPUID
instruction and modifying the values of ebx, ecx,
and edx.

55
VMDETECTGUARD
VMDetectGuard is our solution tool to counter Split
Personality Malware.
 VMDetectGuard runs in two different modes.

VM Guard Mode
 Non VM Guard Mode

56
VMDETECTGUARD
 Output
Generated by VMDetectGuard
Result: Split Personality malware detected/not
detected.
 VM Specific Log
 Instruction Trace
 System Call Trace
 Registry Trace
 Opcode Mix
 Instruction Count
 Diff Tool Feature

57
VMDETECTGUARD
58
RESULTS & ANALYSIS
59
REDPILL
Red Pill is a very well known VM detection tool
by Rutkowska J.
 Runs a single machine language instruction
SIDT and analyses its result.

60
61
SCOOPYNG
 ScoopyNG
is a very well known tool for
VM detection developed by Klein T.
 More reliable tool for VM detection in
comparison to Red Pill.
 It performs the following checks





SIDT check
SLDT check
SGDT check
STR check
IN check (VMware communication channel)
62
63
VMDETECT

This is another well known proof of concept VM
detecting sample that makes use of the VMware
communication channel to detect VMware
Presence.
64
65
BACKDOOR.WIN32.SDBOT.FMN
 Captured
this malware from the
internet.
 Employs Memory check and Timing
Analysis mechanisms .
 In the absence of VMDetectGuard:
“This application cannot run under a
Virtual Machine.”
 In the presence of VMDetectGuard, it
behaved malicious.
66
67
68
VMDETECTGUARD
Running VMDetect in
VirtualPC
Running VMDetect under
masking tool
69
VMDETECTGUARD
Running DetectionChecks
in VirtualBox
Running DetectionChecks
under masking tool
70
OPTIMIZATION
Before (sec)
After (sec)
% decrease in
time taken
VirtualBox
167.310
112.411
32.08%
VirtualPC
294.786
205.953
30.13%
VMware
418.642
299.158
28.54%
Running Firefox binary under masking tool, in all
the three virtual machines.
71
RESULTS

Tested VMDetectGuard
Malwares captured from internet
 Proof of concept tools


The results obtained after testing is given in
table.
72
RESULTS
Binary
Detection Technique Used
Run without tool
Run under tool



Registry Check
File and Process Check
Instruction Check
Detected VirtualBox
Did not detect VirtualBox

File and Process Check
Runs benignly
Runs maliciously



Registry Check
File and Process Check
Invalid Opcode Check
Detected VirtualPC
Did not detect VirtualPC
Backdoor.Win32.SdBot.fmn


File and Process Check
Invalid Opcode Check
Displays a message, “This
application cannot run under a
Virtual Machine
Ran maliciously
VMDetect
Invalid Opcode Check
Detects VirtualPC
Does not detect VirtualPC
Trojen.Karsh-252
Invalid Opcode Check
Displays a message, “This
application cannot run under a
Virtual Machine
Ran Maliciously
Virtual Box
VBDetect: calls others binaries for individual
checks within.
Rebhip
VirtualPC
VPCDetect: calls others binaries for individual
checks within.
73
CONCLUSION
 Split
Personality malware is on a
gradual rise.
 Lack of academic research in this field.
 There does not exist any full-fledged
tool to counter Split Personality
Malware.
 We have designed, implemented and
tested VMwareDetect, a proof of
concept tool that detects the presence
of Vmware.
74
CONCLUSION
We also successfully designed and implemented
VMDetectGuard, a tool to counter Split
Personality malware.
 It detects as well as tricks the split personality
binaries.
 Leads to the effective analysis of malware in the
virtualized environment.
 Increases productivity.

75
SCOPE FOR FUTURE WORK
Further testing of more number of malware.
 Tool is currently built for Vmware, VPC and VB.
 Providing solutions for other analysis tools such
as debugger, sandbox etc.
 The work currently aims at Native binaries



Can be extended to Managed binaries
Extended to other operating systems.
76
REFERENCES


Rutkowska J. (2004). “Red Pill”. http://invisiblethings.org/papers/redpill.html (Nov
20, 2010)
Quist D, Smith V. (2005). “Detecting the Presence of Virtual Machines Using the
Local Data Table”. http://www.offensivecomputing.net/files/active/0/vm.pdf, (Nov 14,
2010)







Klein, T. (2005) “Scoopy Doo” .
http://www.trapkit.de/research/vmm/scoopydoo/index.html (Nov 4, 2010)
P. Ferrie. “Attacks on Virtual Machines”. In Proceedings of the Association of
Virus Asia Researcher Conference, 2007.
Anti-
Zhu D. and Chin E. (2007). “Detection of VM-Aware Malware.”
http://radlab.cs.berkeley.edu/w/uploads/3/3d/Detecting_VM_Aware_Malware.pdf (Dec
1, 2010)
Carpenter M., Liston T., Skoudis E. (2007). "Hiding Virtualization from Attackers
and Malware". IEEE Security and Privacy, June 2007
Lau B, Svajcer V. (2008). “Measuring virtual machine detection in malware using
DSD tracer”. In the Proceedings of Virus Bulletin, 2008
Balzarotti D., Cova M., Karlberger C., Kruegel C, Kirda E, Vigna G. (2010).“Effcient
Detection of Split Personalities in Malware”. In the Proceedings of 17th Annual
Network and Distributed System Security Symposium (NDSS 2010),San Diego,
February 2010
77
REFERENCES








VMware Inc. (2011), “VMware KB: Changing a MAC address in a Windows virtual
machine”.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displa
yKC&externalId=1008473 (Jan 15, 2010)
Pin (2004). “Pin - A Dynamic Binary Instrumentation Tool”. http://www.pintool.org/
(Jan 10, 2010)
Liston T. and Skoudis E. (2006). “On the Cutting Edge: Thwarting Virtual Machine
Detection”.
http://handlers.sans.org/tliston/ThwartingVMDetection_Liston_Skoudis.pdf (Nov 1,
2010)
Tiga, 2007. “Sourpill”,
http://www.woodmann.com/collaborative/tools/index.php/SourPill_VM_Detector (Nov
4, 2010)
VMDetect (2005). “VmDetect, Detect if your program is running inside a Virtual
Machine”. http://www.codeproject.com/KB/system/VmDetect.aspx (Jan 4, 2010)
Guizani, W., Marion, J.-Y., Reynaud-Plantey, D., & Bp, C. S. (2009). “Server-Side
Dynamic Code Analysis”. Analysis, 2009
Omella A. (2006). “Methods for Virtual Machine Detection”. http://www.s21sec.com
(Nov 24, 2010)
OECD, “Malicious Software (Malware): A Security Threat to Internet economy”,
(2007) http://www.oecd.org/dataoecd/53/34/40724457.pdf (Oct 20, 2010)
78
Thank You!
79