CSC 256/456 - Operating Systems - Survey Paper Survey: Virtual Machine Introspection Based System Monitoring and Malware Detection Techniques Haofu Liao Department of Computer Science University of Rochester haofu.liao@rochester.edu Abstract and attacker. Once the hosts have been compromised, the host-based security tools are highly susceptible to attack. To increase attack resistance, a good idea is to move the security tools outside the hosts. The major problem of such approach is the lost of visibility. For example, in intrusion detection, the security systems are usually placed in network. Thus, they cannot monitor the internal host states or events. All the information they need must be collected from network traffic which gives the attacker more room to hide outside the view of the security systems. Hence, a security tool that offers both high visibility and attack or tamper resistance is required. To this end, a technique called virtual machine introspection was introduced in 2004 [6]. In the pioneer work of this area, the authors proposed a novel architecture, Livewire, to build a intrusion detection system (IDS) using VMM. They put their IDS into a security VM and inspect the guest VM with the help of VMM. Livewire leverages three properties of the VMM. First, a software that runs in a virtual machine is isolated from the VMM. Second, VMMs can access all the states of a virtual machine. Third, VMMs can also interpose on certain virtual machine operations. Therefore, a VMI-based architecture can provide a good visibility as well as a high attack resistance to the IDS. Inspired by Livewire, several other VMI-based system monitoring and malware detection techniques are proposed. In 2005, a security system called IntroVirt was proposed [8]. The system uses introspection and replay to test if a system was previously attacked through a known vulnerability. In 2007, Jiang et al. [7] presented a VMI-based stealthy malware detection technique, VMWatcher. It improves the visibility of the security tools by restoring the internal semantic objects (e.g., files and directories) of a VM from the outside in a native and non-intrusive manner. The same year, Payne et al. [12] built a VMI-based monitoring library called XenAccess. Their contribution is considered important for the reason that the XenAccess monitoring library satisfies a series of high-level requirements and delivers secure and efficient functionality to VMI-based techniques. What’s more, In recent years, modern malware are growing powerful. It is very common to see them subvert their victim machine’s security tools upon installation. Traditionally, one can solve this problem by moving critical security services into network so that they are isolated from monitored host and attackers. However, this will result in a poor review of what’s happening inside the host. To address this problem, a technique called virtual machine introspection (VMI) is proposed. By placing security services into virtual machines (VMs), it inspects the host from outside for the purpose of analyzing the software running inside it. This technique leverages virtual machine monitor (VMM) technology such that the hardware state of the host can be directly inspected. Because of the abstraction and isolation provided by virtualization, VMI has received significant discussion in research literature. This paper introduces four popular VMIbased techniques in the field and gives a comparative study on these techniques. Keywords Malware Analysis, Intrusion Detection, Virtual Machine Monitor, Virtual Machine Introspection 1. Introduction Host-based security tools such as anti-malware or intrusion detection system can inspect the monitored system directly. Enjoyed by the native, semantic view (e.g., files, processes, and kernel modules) of the hosts, host-based security tools are much easier to detect malware and attacks. However, traditional host-based security tools are not able to satisfy the high security requirement of today’s computer systems. Modern internet malware (e.g. rootkits and bots) often try to detect and subvert existing security tools as long as they get installed on the host. The problem of the host-based security tools is that they run inside the very hosts they are protecting which makes them exposed directly to malware 1 2015/12/11 2.2 another technique, Lares was introduced in 2008 [11]. The authors implemented a VM isolated security system that has the ability to do the active monitoring at a higher semantic level. The rest of this paper is organized as follows. Section 2 introduces some preliminaries for this paper and provides a detailed introduction to four important VMI-based techniques. Section 3 compares those techniques side by side and section 4 presents the summary. 2. 2.2.1 Thread Model and Assumptions Almost all the VMI-based techniques make similar assumptions to their thread model [6–8, 11]. First, the attacker or malware can compromise the monitored host arbitrarily and the code running inside may be totally malicious. Second, the VMM and the security VM (if needed) that runs the security tools are part of the trusted computing base. It means the attacker or malware can’t corrupt the security VM and the underlying VMM. This assumption is based on the fact that the interface and protection model of a VMM is significantly simpler than a modern operating system. What’s more, the authors of [11] also assume that the machine undergoes a secure boot [1]. This should hold for other VMIbased techniques as well. Techniques In this section, we first introduce the some terms that will be used throughout the paper. Then we discuss the prerequisites for the VMI-based techniques. More specifically, we present the typical thread model and assumptions a VMI-based technique should have in their design. We also discuss the standard requirements that VMI-based monitoring library should consider to support VMI-based techniques. XenAccess is introduced as an example. Finally, we introduce the four popular VMI-based techniques: Livewire, IntroVirt, VMwatcher, and Lares. 2.1 Prerequisites 2.2.2 Monitoring Library Requirements The VMI-based techniques usually need to set up a secure VM that is used to monitor the other VMs (target) running on the same VMM. A good VMI-based technique requires researchers to properly build a monitoring architecture around the VMM, the secure VM and the target VMs. A proper monitoring architecture should be secure, flexible and robust so that the VMI-based security systems built on top of it are not likely for an intruder to compromise, evade or disable the monitors. Driven by the question of how to build a proper monitoring architecture for VMI-based techniques, authors of [12] proposed the six high-level requirements: Terminology Virtualization and Virtual Machine (VM) Virtualization refers to a technology which allows one or more operating systems to run as applications within other operating systems [15]. A virtual machine is an instance of such application. 1. “No superfluous modifications to the VMM. ” Virtual Machine Monitor (VMM) The set of computer software and hardware that creates, runs and manages virtual machines. There are two types of VMMs. The Type I VMM runs directly on the hardware and the Type II runs on a conventional operating system [15]. A VMM is also referred as a hypervisor. 2. “No modifications to the VM or the target OS. ” 3. “Small performance impact. ” 4. “Rapid development of new monitors. ” 5. “Ability to monitor any data on target OS. ” 6. “Target OS cannot tamper with monitors. ” Vulnerability “A software vulnerability is an instance of an error in the specification, development, or configuration of software such that its execution can violate the security policy.” [10] With these requirements in mind, they presented the XenAccess monitoring library which provides virtual memory introspection and virtual disk monitoring capabilities. They build XenAccess on top of the Xen which is a Type I VMM and runs directly on the hardware. Xen’s infrastructure fits so well with the authors’ monitoring needs that it is unnecessary to modify the VMM. Thus, the requirement 1 is satisfied. Similarly, working with Xen’s infrastructure XenAccess does not need to make any changes to the target OS. So, the requirement 2 is satisfied. The authors also place the monitors in a different VM than the target OS. Thus, the target OS can’t tamper with the monitors and requirement 6 is satisfied. If more explicit isolation is required, the builtin mandatory access control in the Xen VMM [13] can also be used. Though XenAccess architecture only provides the introspection ability for memory and disk I/O, one can easily extend the monitoring to other hardware states with Xen. Predicate A predicate is a program module that tests the state of a target system as it executes and detects when the vulnerability is triggered [8]. Trusted Computing Base (TCB) The set of hardware and/or software components that are necessary to enforce all the security rules [16]. Intrusion Detection System (IDS) A device or software application that monitors attempted or successful intrusions into computer systems or other security violations, and to initiate appropriate responses to the intrusions or security violations [3, 15]. 2 2015/12/11 host by collecting hardware state from the VMM. The OS interface library then interprets the hardware-level state into OS-level semantics so that they can be used by the policy engine. The policy engine sends queries to the OS interface library and determines the real state of the host using the built-in policy modules. Based on the real state of the host, the policy engine can send commands to the VMM over the VMM interface. The VMM will then control or inspect the host according to the information in the commands. The Livewire architecture provides additional functionality by leveraging existing VMM mechanism which satisfies the requirement 1. It also makes no changes to the monitored host, so it satisfies the requirement 2. To improve the performance, Livewire only traps events that would imply definite misuse. According to the performance results, we find that no measurable overhead is imposed to Livewire. Thus, requirement 3 is satisfied. Due to its ability to reconstruct the OS-level semantics, Livewire can monitor any data it wants on the target OS. So, requirement 5 is also satisfied. Livewire does not directly communicate with the monitored host and the only way it can get information from the host is through the VMM. Thus, the host can’t tamper with Livewire and requirement 6 is satisfied. IDS Policy Engine Policy Modules Monitored Host Command Guest Apps Guest OS Virtual Machine Callback or Respose Policy Framework Query Response Config File Guest OS Metadata OS Interface Library Hardware State Virtual Machine Monitor Figure 1: The architecture of Livewire [6]. Thus, requirement 5 is also satisfied. satisfied. The experiments of XenAccess’s monitoring performance show that the overhead added by XenAccess is minimal. Thus, requirement 3 is satisfied. Compared to other virtualization monitoring architectures, such as the work by Asrigo et al. [2], XenAccess’s monitors are small user-space applications. Hence, the monitors can be developed easily and requirement 4 is satisfied. 2.3 2.4 IntroVirt IntroVirt is another VMI-based intrusion detection system. However, unlike Livewire which mainly monitors malicious activities or policy violations, IntroVirt focuses on security vulnerabilities. System administers often raise two questions when a vulnerability is discovered in their system. First, they want to know whether their system has been attacked through the newly discovered vulnerability. Second, they also want to know how can they detect and respond to intrusions before a patch is applied to their system. Driven by these two questions, the authors of IntroVirt designed a system that executes vulnerability-specific, perturbationfree predicates as the system runs or replays. [8] IntroVirt can perfectly solve the two problems encountered by system administers. First, IntroVirt runs with the User-Mode Linux (UML) [4] VMM that supports the virtual-machine replay functionality [5, 9]. By combining this functionality with the vulnerability-specific predicates, IntroVirt is able to detect whether some vulnerability was exploited before. [8] Second, with the help of vulnerability-specific predicates, IntroVirt implements some response strategies to alert or prevent the known-but-unpatched vulnerabilities from being exploited by attackers [8]. Figure 2 illustrates the architecture of IntroVirt. As UML is a Type II VMM, both the IntroVirt system and the target system run on a host operating system. For demonstration purpose, the target system is divided into two parts: one is the guest kernel and the other is the guest applications. Similar with other VMI-based systems, IntroVirt executes its predicate engine on another VM. However, the design consideration is different. In this system, the authors focus on Livewire Livewire is the first VMI-base architecture in this field. The authors presents this intrusion detection system (IDS) using VMM. They put the IDS into a hardware domain protected by VMM to isolate the host with IDS. The high-confidence barrier provided by VMM makes it very hard for the attacker or the malware to compromise the IDS [6]. The VMM also provides Livewire the ability to inspect the hardware state of the VM that the monitored host is running on. Using the the hardware-level state (e.g. physical memory and registers) and combining the knowledge of the operating system structures inside the VM, Livewire is able to reconstruct OS-level semantics. It uses them to analysis the behavior of monitored host. Because the VMM has privileged access to the monitored host, the authors designed a VMM interface for Livewire. Livewire sends commands to the VMM over this interface so that it can monitor and control the host indirectly. Figure 1 illustrates the architecture of Livewire. The block on the left is the VM that runs the monitored host. The block on the right is the VMI-based IDS which contains two parts: the policy engine and the OS interface library. The IDS and the VM run on the top of the VMM. They are fully isolated from each other and from the VMM. The IDS monitors the 3 2015/12/11 Predicate Engine Guest-User Host Process Virtual Machine Guest-Kernel Host Process Guest Applications User-Mode Linux + IntroVir Infrastructure AntiVirus Software Host Operating System Host Operating System the requirement that predicates must not perturb the target state. This is because if the predicate is allowed to perturb the target state, a buggy predicate might break the system. The isolation provided by VMI perfectly solves this problem. The same as other VMI-based techniques, the predicate engine leverages the hardware state provided from the VMM and combines the knowledge of the target OS structure to reconstruct the system-level states. It uses these states to monitor the discovered vulnerable area in the target system. IntroVirt makes no changes to the VMM since it only uses the low-level functions provided by the VMM. Requirement 1 is satisfied. IntroVirt runs in a perturbation-free fashion. Thus, no modifications to the VM or the target OS is required which satisfies requirement 2. According to the performance results from [8], IntroVirt has 15%-76% overhead in its execution. This is relatively high. Hence, the small performance impact requirement (requirement 6) is not satisfied. As IntroVirt can reconstruct all the data it wants from the target system, and it is fully isolated from the target system, requirement 5 and 6 are satisfied. Figure 3: The architecture of VMwatcher [7]. VM to the “out-of-the-box” anti-malware software natively. Hence, the anti-malware software can scan and detect malicious code and processes in the protected VM without knowing its existence. Moreover, since the host and guest may run on different platforms, it is possible to perform crossplatform malware detection, which means one can use the anti-malware designed for one platform (e.g. windows) to detect malicious programs on another platform (e.g Linux) [7]. Figure 3 shows the architecture of VMwatcher. In this architecture, VMwatcher and the VMM run on the top of a host operating system. Although this means a type II VMM is in use, VMwatcher can be easily implemented on type I VMMs as well. The guest OS runs on a VM that is fully isolated from VMwatcher. VMwatcher monitors the guest OS with the hardware states collected from the VMM. Using the guest view casting technique, it can interpret the hardware states into a high-level semantic view. All these semantics are presented to the host operating system natively. It means any process on the host operating system can use them without knowing that they are actually from the protected VM. Thus, an anti-malware software can scan and detect possible infections of the guest OS exactly the same way as it does for the host OS. In this case, the anti-malware software designed for the host OS is actually used for the guest OS. With a quick examination, we can find that VMwatcher does no superfluous modifications to the VMM and no modifications to the VM or the guest OS, and the guest OS does not tamper with VMwatcher. Hence, requirement 1, 2, and 6 are satisfied. Thanks to the guest view casting technique, VMwatcher perfectly satisfies requirement 5. Since the virtualization will introduce some potential disk I/O slowdown and the host usually has larger memory space, VMwatcher can actually outperform the guest OS in malware scanning time [7]. Hence, requirement 3 is also satisfied. VMwatcher VMwatcher is a VMI-based malware detection system. Similar with other VMI-based system, VMwatcher leverages the VMM to conduct malware detection outside the protected VM so that it can achieve high tamper-resistant. Unlike other VMI-based system, the authors designed a dedicated technique called guest view casting which not only can reconstruct the semantic-level view of the VM but also can restore and present the high-fidelity semantic objects to antimalware softwares outside the guest. Their malware detection system has two unique capabilities: First, VMwatcher can perform a comparison-based malware detection. With the ability of viewing the protected VM from both inside and outside (of the VM), VMwatcher compares the two different views of the same objects of interest and detects the hidden malware by discovering the inconsistency of the two. Second, VMwatcher can perform “out-of-the-box” 1 execution using off-the-shelf anti-malware software for the protected VM [7]. With the help of the guest view casting technique, VMwatcher presents the internal view of the protected 1 Outside Guest Operating System VMM + VMwatcher Infrastructure Figure 2: The architecture of IntroVirt [8]. 2.5 VMwatcher the protected VM. 4 2015/12/11 Security VM Guest VM User Processes Introspection API Security Application Hooks … Memory Protector Trampoline Security Drive Hypervisor Hardware (CPU + Virtualization Extension) Figure 4: The architecture of Lares [11]. 2.6 Lares ning in the security VM [11]. Lares makes sure that the hooks and trampoline are small and self-contained. Thus, a write-protection to their memory footprint is enough to ensure their correct behavior. This memory protection functionality is implemented inside the VMM. The communication between VMs is supported by the VMM. It makes sure the trampoline can send events captured from the guest VM to the security driver in the security VM, and the replies sent from the security driver can also be delivered to the trampoline [11]. Lares puts all its back-end components to the security VM. The security application makes decision and sends response according to the event that was trapped. The security driver relays events between the trampoline and the security application. The introspection API is used by the security application to implement the introspection. It allows the security application to gather information from the trapped events. Unlike other VMI-based techniques, to implement the active monitoring functionality, Lares must modify the guest OS. So, requirement 2 is not satisfied. Requirement 1 still holds, since Lares only makes necessary changes to the VMM to make sure the active monitoring is secure. Requirement 5 also holds because Lares can reconstruct all the OS semantics it needs for the active monitoring. By keeping the hooks and the trampoline self-contained and small, Lares is able to use the memory protection mechanism to make sure OS cannot tamper with monitors. Thus, requirement 6 holds. The switching between the guest VM, the hypervisor, and the secure VM has a large overhead [14]. Hence, 5 is not satisfied. Lares is a VMI-based active monitoring security architecture. Most of the VMI-based security tools focus on passive monitoring. They only scan and poll the target system externally [11]. These security tools gain benefits from passive monitoring due to its low overhead and perturbationfree features. However, the big limitation of passive monitoring is its inability to guarantee interposition on events before they happen. To address this problem, the authors of Lares proposed a secure, active monitoring architecture for VMI-based techniques. In active monitoring, the security tool places hooks inside the monitored system. When the execution reaches a security-critical area, these hooks will be activated and the control will be passed to the security tool [11]. Hence, when a malicious activity is about to happen, it will invoke a trap to the security tool so that the security tool can stop the malicious process before it really does any damage to the protected system. However, the active monitoring requires that some security-critical code must be placed inside the untrusted VMs [11]. It means that part of the security facility will still be exposed to attackers even though VMI is in use. Therefore, to address this problem and avoid the exposure, Lares limits the functionality of the code placed inside the untrusted VM and provides specialized protection mechanisms to ensure the security of active monitoring. The overall structure of the Lares architecture is illustrated in Figure 4, where the components in gray represent the TCB. Lares includes two VMs: the untrusted guest VM and a security VM that is part of our TCB. They both run on the top of a type I VMM (Hypervisor). To gain the isolation benefit that comes with VMI and minimize the exposure to the attacker, Lares puts most of its security facilities in the security VM and only keeps the hooks and a trampoline in the untrusted guest VM. Here, the trampoline is carefully-crafted light-weighted program that supports communication between the hooks and the security driver run- 3. Comparison The comparison of the four techniques is given in Table 1. Both Livewire and IntroVir are intrusion detection systems. However, the later has a vulnerability detection feature. More precisely, it tries to detect whether a vulnerability is exploited before and after its disclosure. VMwatcher is a 5 2015/12/11 Architecture Application Monitoring Type VMM Type Visibility Overhead Livewire (2003) Intrusion Detection Passive Type I OS-level Light IntroVir (2005) Intrusion Detection Passive Type II OS-level Medium VMwatcher (2007) Malware Detection Passive Type II Native Light Lares (2008) Any system that uses active monitoring Active Type I OS-level Heavy Table 1: Architecture comparison of VMI-based techniques. malware detection system. It enables an off-the-shelf antivirus software on the host OS to scan and detect malware on the guest OS. Lares is a general architecture that provides secure active monitoring functionality. The first three architectures only use passive monitoring due to their application requirement. Lares uses active monitoring by nature. Any applications that require interposition of events before they happen can benefit from the extra security provided by Lares. Livewire and Lares use Type I VMMs. IntroVir and VMwatcher use Type II VMMs. For VMwatcher, even though it is illustrated using Type II VMM, it actually supports Type I VMMs as well. For the visibility, Livewire, IntroVir and Lares are capable of reconstructing OS-level semantics from protected VM. This is the basic property of VMI-based systems. VMwatcher can go beyond this property due to its guest view casting technique. It can provide native view of the guest OS for the host anti-virus software. For the overhead, both Livewire and VMwatcher have no measurable overhead due to their simplicity. IntroVir has medium overhead because it uses the UML VMM which is not so efficient. Lares doesn’t have a good performance because of the active monitoring. It requires constantly switching between the guest VM, the hypervisor, and the secure VM which generates large overhead. 4. [2] A SRIGO , K., L ITTY, L., AND L IE , D. Using vmm-based sensors to monitor honeypots. In Proceedings of the 2nd international conference on Virtual execution environments (2006), ACM, pp. 13–23. [3] D ENNING , D. E. An intrusion-detection model. IEEE Trans. Softw. Eng. 13, 2 (Feb. 1987), 222–232. [4] D IKE , J. A user-mode port of the linux kernel. In Proceedings of the 4th Annual Linux Showcase & Conference - Volume 4 (Berkeley, CA, USA, 2000), ALS’00, USENIX Association, pp. 7–7. [5] D UNLAP, G. W., K ING , S. T., C INAR , S., BASRAI , M. A., AND C HEN , P. M. Revirt: Enabling intrusion analysis through virtual-machine logging and replay. SIGOPS Oper. Syst. Rev. 36, SI (Dec. 2002), 211–224. [6] G ARFINKEL , T., AND ROSENBLUM , M. A virtual machine introspection based architecture for intrusion detection. In In Proc. Network and Distributed Systems Security Symposium (2003), pp. 191–206. [7] J IANG , X., WANG , X., AND X U , D. Stealthy malware detection through vmm-based ”out-of-the-box” semantic view reconstruction. In Proceedings of the 14th ACM Conference on Computer and Communications Security (New York, NY, USA, 2007), CCS ’07, ACM, pp. 128–138. [8] J OSHI , A., K ING , S. T., D UNLAP, G. W., AND C HEN , P. M. Detecting past and present intrusions through vulnerabilityspecific predicates. In Proceedings of the Twentieth ACM Symposium on Operating Systems Principles (New York, NY, USA, 2005), SOSP ’05, ACM, pp. 91–104. Summary We have introduced the background and history of VMI and its related techniques. We’ve also discussed the thread model and implementation requirements for VMI-based techniques. More importantly, we’ve studied four popular VMI-based techniques in detail and compared their characteristics one by one. VMI can provide the security systems with high visibility as well as high attack resistance. It can be used in wide range of applications. The four techniques show that with the help of VMI, their systems can be much more secure in their specialized area. VMI brings no measurable overhead and functionality lost in most of the techniques. [9] K ING , S. T., D UNLAP, G. W., AND C HEN , P. M. Debugging operating systems with time-traveling virtual machines (awarded general track best paper award!). In Proceedings of the 2005 USENIX Annual Technical Conference, April 10-15, 2005, Anaheim, CA, USA (2005), pp. 1–15. [10] K RSUL , I. V. Software Vulnerability Analysis. PhD thesis, West Lafayette, IN, USA, 1998. AAI9900214. [11] PAYNE , B., C ARBONE , M., S HARIF, M., AND L EE , W. Lares: An architecture for secure active monitoring using virtualization. In Security and Privacy, 2008. SP 2008. IEEE Symposium on (May 2008), pp. 233–247. References [12] PAYNE , B., DE C ARBONE , M., AND L EE , W. Secure and flexible monitoring of virtual machines. In Computer Security Applications Conference, 2007. ACSAC 2007. Twenty-Third Annual (Dec 2007), pp. 385–397. [1] A RBAUGH , W., FARBER , D., AND S MITH , J. A secure and reliable bootstrap architecture. In Security and Privacy, 1997. Proceedings., 1997 IEEE Symposium on (May 1997), pp. 65– 71. [13] S AILER , R., JAEGER , T., VALDEZ , E., C ACERES , R., 6 2015/12/11 P EREZ , R., B ERGER , S., G RIFFIN , J., AND VAN D OORN , L. Building a mac-based security architecture for the xen open-source hypervisor. In Computer Security Applications Conference, 21st Annual (Dec 2005), pp. 10 pp.–285. [14] S HARIF, M. I., L EE , W., C UI , W., AND L ANZI , A. Secure in-vm monitoring using hardware virtualization. In Proceedings of the 16th ACM Conference on Computer and Communications Security (New York, NY, USA, 2009), CCS ’09, ACM, pp. 477–487. [15] S ILBERSCHATZ , A., G ALVIN , P. B., AND G AGNE , G. Operating System Concepts, 8th ed. Wiley Publishing, 2008. [16] TANENBAUM , A. S. Modern Operating Systems, 3rd ed. Prentice Hall Press, Upper Saddle River, NJ, USA, 2007. 7 2015/12/11