Live migration of Virtual Machines


Live migration of Virtual


Nour Stefan, SCPD

• Introduction

• Related work

• Design

• Writable Working Sets

• Implementation Issues

• Evaluation

• Future work

• Conclusions


• OS virtualization

– Data centers

– Cluster computing

• Live OS migration

– Avoid problem of “residual dependencies”

– In-memory state can be transferred in a consistent and efficient way

• Kernel-internal state

• Application-level state

– Separation of concerns between Users and Operator of a data center or cluster

– Separation of hardware and software considerations, and consolidating clustered hardware into a single coherent management domain

• High-performance migration support for Xen

Related work

• Collective project

– For slow connections and longer time spans

– Stop the OS execution while transfer

• Zap

• NomadBIOS

– Pre-copy migration

– Not adapting to the writable working set


• Migrating memory

• Balancing Downtime and Total migration time

– Push phase

– Stop-and-copy phase

– Pull phase

• Local resources

– Connections to local devices(disks , network interfaces)

– Single switched LAN

– Generate an unsolicited ARP reply from migrated host, advertising that the IP has moved to a new location

– Network-Attached Storage

Writable Working Sets

Measuring Writable Working Sets

Measuring Writable Working Sets

Measuring Writable Working Sets

Implementation Issues

• Managed migration

– Performed largely outside the migratee

– Migration daemons running in the management VM of the source and destination (new VM on destination)

– Rounds of copying (dirtied during the previous round)

– Dirty bitmap copied from Xen at start of each round

– Shadow page tables (read-only page-tables entries => page fault trapped by Xen)

Implementation Issues

• Self migration

– Implemented within the migratee OS

– Migration stub on destination machine

– Consistent OS checkpointing

• Two-stage stop-and-copy phase

– Disables all OS activity except for migration => final scan of dirty bitmap => shadow buffer

• Transfer shadow buffer

Implementation Issues

• Dynamic Rate-Limiting

• Rapid Page Dirtying


• Test setup

– Dual Intel Xeon 2GHz CPU and 2GB memory

– TG3 broadband





Future work

• Cluster management

• Wide Area Network redirection

• Migrating Block Devices


• By integrating live OS migration into the Xen virtual machine monitor we enable rapid movement of interactive workloads within clusters and data centers.

Our dynamic network-bandwidth adaptation allows migration to proceed with minimal impact on running services, while reducing total downtime to below discernable thresholds.

• Our comprehensive evaluation shows that realistic server workloads such as SPECweb99 can be migrated with just 210ms downtime, while a Quake3 game server is migrated with an imperceptible 60ms outage.