Live migration of Virtual
Machines
Nour Stefan, SCPD

• Introduction
• Related work
• Design
• Writable Working Sets
• Implementation Issues
• Evaluation
• Future work
• Conclusions

Introduction
• OS virtualization
– Data centers
– Cluster computing
• Live OS migration
– Avoid problem of “residual dependencies”
– In-memory state can be transferred in a consistent and efficient way
• Kernel-internal state
• Application-level state
– Separation of concerns between Users and Operator of a data center or cluster
– Separation of hardware and software considerations, and consolidating clustered hardware into a single coherent management domain
• High-performance migration support for Xen

Related work
• Collective project
– For slow connections and longer time spans
– Stop the OS execution while transfer
• Zap
• NomadBIOS
– Pre-copy migration
– Not adapting to the writable working set

Design
• Migrating memory
• Balancing Downtime and Total migration time
– Push phase
– Stop-and-copy phase
– Pull phase
• Local resources
– Connections to local devices(disks , network interfaces)
– Single switched LAN
– Generate an unsolicited ARP reply from migrated host, advertising that the IP has moved to a new location
– Network-Attached Storage

Writable Working Sets

Measuring Writable Working Sets

Measuring Writable Working Sets

Measuring Writable Working Sets

Implementation Issues
• Managed migration
– Performed largely outside the migratee
– Migration daemons running in the management VM of the source and destination (new VM on destination)
– Rounds of copying (dirtied during the previous round)
– Dirty bitmap copied from Xen at start of each round
– Shadow page tables (read-only page-tables entries => page fault trapped by Xen)

Implementation Issues
• Self migration
– Implemented within the migratee OS
– Migration stub on destination machine
– Consistent OS checkpointing
• Two-stage stop-and-copy phase
– Disables all OS activity except for migration => final scan of dirty bitmap => shadow buffer
• Transfer shadow buffer

Implementation Issues
• Dynamic Rate-Limiting
• Rapid Page Dirtying

Evaluation
• Test setup
– Dual Intel Xeon 2GHz CPU and 2GB memory
– TG3 broadband

Evaluation

Evaluation

Evaluation

Evaluation

Future work
• Cluster management
• Wide Area Network redirection
• Migrating Block Devices

Conclusions
• By integrating live OS migration into the Xen virtual machine monitor we enable rapid movement of interactive workloads within clusters and data centers.
Our dynamic network-bandwidth adaptation allows migration to proceed with minimal impact on running services, while reducing total downtime to below discernable thresholds.
• Our comprehensive evaluation shows that realistic server workloads such as SPECweb99 can be migrated with just 210ms downtime, while a Quake3 game server is migrated with an imperceptible 60ms outage.