Virtualization in Mobile Systems
Mahadev Satyanarayanan
School of Computer Science
Carnegie Mellon University
Based on results and insights from recent collaborative research with:
Steve Smaldone, Adam Wolbach, Benjamin Gilbert, Jan Harkes, Nilton Bila, Sarah Rutlidge, Liviu
Iftode, Eyal de Lara, Andres Lagar-Cavilla, Victor Bahl, Ramon Caceres, Nigel Davies, Roy Want
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
1
Sad Reality of Mobile Computing
Hardware limitations
• vs. static elements of same era
• weight, power, size constraints
• CPU, memory, disk, display, 
Wireless communication uncertainty
• bandwidth / latency variation
• intermittent connectivity
• may cost real money
Finite energy source
• actions may be slowed or deferred
• communication costs energy
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
2
What Has Changed?
Previous slide was true 15+ years ago (early 1990s)
• huge hardware and wireless networking improvements since
• but deep essentials haven’t changed
• on autopilot, same slide will be true 15+ years hence (2020)
Resource poverty is the enduring attribute of mobile computing
• How can we change this?
• fundamental paradigm shift
How can we escape the resource trap?
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
3
Escaping the Resource Trap
Today’s mobility metaphor:
Self-sufficient but resource-poor
New mobility metaphor:
Leveraged and resource-rich
Leverage the Cloud!
(but keep the Swiss Army Knife as fallback)
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
4
Leveraging the Cloud - I
The Legacy PC World
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
5
2009 ACM Turing Award
Chuck Thacker
Pioneer Honored for Design of
First Modern Personal
Computer and Other Major
Innovations
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
6
Why Was the PC Invented?
The nicest thing about the Alto is that it doesn’t run
faster at night
(J. Morris) from
A similar lesson was learned about processor time.
With interactive use the response time to a demand
for computing is important, since a person is
waiting for it. Many attempts were made to tune the
processor scheduling as a function of priority of the
computation, working set size, memory loading,
past history, likelihood of an i/o request, etc; these
efforts failed.
“Hints for
Computer System
Design”
by Butler Lampson
SOSP 1983

The natural extension of this strategy is the
personal computer, in which each user has at least
one processor to himself.
WhatFrom
we gained: unvarying crisp interaction
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
7
What We Gave Up: Mobility
Personal computing involved a tradeoff
• we won big on one set of issues (crisp interaction + usability)
• we gave up on a second (seamless mobility + easy adminstration)
Today, one’s PC is a personal fortress
• complete computing world of its own
uniquely customized by you
long, slow setup process to get every setup detail right
• discourages use of pervasive hardware
ok for stateless apps like web browsing
not ok for personal productivity apps
• expensive system administration
Can we regain what we gave up 25+ years ago?
• enjoy seamless mobility across pervasive hardware?
• trivial system administration at edges
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
8
VM-based Transient PC Model
VM image
(memory, disk, etc.)
+ additional meta-data
Cloud
Storage
Parcel
Parcel
“Your own PCanywhere, anytime”
Proposed in WMCSA June 2002
“Internet Suspend/Resume”, Kozuch and Satyanarayanan
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
9
ISR Evolution
ISR-1
ISR-2
late 2001 – early 2002
Performance tradeoff exploration
Proof of concept
• VMM = VMware Workstation 3.0
•
•
•
•
• dist. storage = NFS
• copyin/copyout of entire VM state
OpenISR
mid 2006 – now (release 0.9.9)
early 2002 – late 2004
VMM = VMware Workstation 3.0
dist. storage = Coda
loadable kernel module (“fauxide”)
user-level ISR client (“vulpes”)
ISR-3
late 2004 – mid 2006
Production-quality open source system
Pilot deployment
VMM-agnostic (VirtualBox, KVM, VMware, )
complete re-engineering of code base
total rewrite of kernel code (nexus)
ongoing evolution of client and server
functionality and performance
• continuous deep use (~15-20 users)
• PocketISR boot capability
•
•
•
•
•
•
•
•
•
•
•
© 2009-2010 M. Satyanarayanan
VMM = VMware Workstation 4.5
dist. storage = user-level chunk store
major vulpes changes
23 users, peak usage Jan-Aug 2005
extensive analysis of usage data
predicted high value of CAS
revealed fatal flaw in client design
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
10
Using Mobile Devices with ISR
ISR disadvantage: variable user experience
(Suspend & Resume delays depend on Internet connectivity)
Can smartphones improve ISR user experience?
• device that people already carry
• ample storage + multiple modes of connectivity
• very small mobility footprint
Embodied in experimental Horatio extension to OpenISR
(Smaldone et al, MobiSys 2009)
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
11
Types of Cloud Computing
VM Storage
(e.g. Grid Computing)
(e.g. Amazon EC2)
Classic PC model
Transient PC
(e.g. laptops)
(e.g. ISR)
better availability
more comute power
Cloud
Remote execution Managed execution
local-local
© 2009-2010 M. Satyanarayanan
cloud-cloud
Local
VM Execution
local-cloud
Cloud
crisper interaction
Local
cloud-local
better safety
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
12
Leveraging the Cloud - II
New Resource-Rich Mobile Apps
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
13
Machine Translation Today
0.85
0.8
Human Scoring Range
0.7289
0.7447
BLEU SCORES
0.7
0.6
0.5551
0.5610
Systran
Spanish
SDL
Spanish
0.5137
0.5
0.4
0.3859
0.3
Google
Google
Chinese
Arabic
(‘06 NIST) (‘05 NIST)
CBMT
Google
Spanish
Spanish
’08 top lang
Based on same Spanish test set
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
14
Face Recognition Today
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
15
What’s The Catch?
These are resource-intensive applications
• State-of-art performance and quality only with room full of servers
• How do we achieve this “in the wild”?
(on resource-poor, energy-limited mobile hardware)
Obvious solution: leverage the cloud!
But your cloud may be far away 
End-to-end latency matters for crisp interaction
• e.g., real-time two-way language translation on mobile devices
• e.g, augmented reality for cognitive assistance via “smart glasses”
•  and many other examples
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
16
Latency Hurts Even If Bandwidth Good
(E.g. QuakeViz interactive benchmark on VNC thin client 100 Mbps)
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
17
Sample Internet2 RTTs (milliseconds)
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
18
Latency on 3G Networks
“The wireless delay in the 3G network dominates the whole
network path delay, e.g., latency to the first pingable hop is around
200ms, which is close to the end-to-end Ping latency to landmark
servers distributed across the U.S.”
from “Anatomizing Application Performance Differences on
Smartphones”, to appear in MobiSys 2010 (Huang et al)
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
19
Solution: Create a Tiny Cloud Nearby
Olympus
Mobile Eye Trek
Wearable
Computer
Android
Phone
WAN to
distant cloud
on Internet
Low-latency
high-bandwidth
1-hop wireless
network
Nokia N810
Tablet
Handtalk
Wearable
Glove
© 2009-2010 M. Satyanarayanan
Coffee shop
Cloudlet
cloudlet = (compute cluster
+ wireless access point
+ wired Internet access
+ no battery limitations)
 “data center in a box”
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
20
Local Wireless Bandwidth
Original motivation for cloudlets was latency
But 1-hop wireless bandwidth to cloudlet also a win
• wireless LAN bw typically 100X wireless WAN bw
e.g. 802.11n ≈ 400 Mbps but HSPDA ≈ 2 Mbps
• shipping large objects within interactive time bounds
e.g. captured images in an augmented reality system
4MB JPEG image takes 80 ms @ 400 Mbps, but 16 seconds @ 2 Mbps
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
21
Cloudlet vs. Cloud
Cloudlet
Cloud
State
Only soft state
Hard and soft state
Management
Appliance model:
self-managed; little
professional attention
Utility model:
professionally administered,
24x7 operator coverage
Environment
“Data center in a box” at
customer premises
Machine room with power
conditioning and cooling
Ownership
Decentralized ownership
by local business
Centralized ownership by
Amazon,Yahoo!, etc.
Network
LAN latency and
bandwidth
Internet latency and
bandwidth
Sharing
Few users at a time
100s to 1000s of users
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
22
Key Challenges
1. Trusting infrastructure
•
tamper-resistant hardware (“first-world infrastructure”)
•
portable device as root of trust (e.g TrustSniffer)
2. Finding the exactly right software on it
uniformity  deployer value
specificity  end-user value
© 2009-2010 M. Satyanarayanan
inherent tension
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
23
Transient Customization
Deliver fully configured virtual machine (VM) to infrastructure
Problem: too large, too slow for transient use
Solution: assemble VM on the fly  dynamic VM synthesis
• prefetch large, relatively static, widely-used piece (“base VM”)
• deliver small patch (“VM overlay”) just before use
• discard VM after use
VM overlay can come from
• mobile device over wireless link, or
• web site under control of mobile device (URL and decryption key)
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
24
Dynamic VM Synthesis
Preload base VM
M
o
b
i
l
e
D
e
v
i
c
e
Discover & negotiate
use of cloudlet
(base + overlay)  launch VM
Execute launch VM
Use
cloudlet
user-driven
device-VM
interactions
C
l
o
u
d
l
e
t
Finish use
Create VM residue
Depart
© 2009-2010 M. Satyanarayanan
Discard VM
Optional: cache VM overlay
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
25
Typical Overlay Sizes
(base VM = 8GB Ubuntu Linux)
Application
Compressed
VM Overlay
Size (MB)
Uncompressed
VM Overlay
Size (MB)
Install Package
Size (MB)
AbiWord
119.5
364.2
10.0
GIMP
141.0
404.7
16.0
Gnumeric
165.3
519.8
16.0
Kpresenter
149.4
426.8
9.1
PathFind
196.6
437.0
36.8
SnapFind
63.7
222.0
8.8
Null
5.9
24.8
0.0
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
26
VM Synthesis Time at 100Mbps
(untuned proof-of-concept prototype)
Other
140
Resume VM
Largest standard deviation is 5.3% of mean
120
Apply VM overlay
Nearly half the total
All in the infrastructure
Potentially optimizable
Decompress VM overlay
Transfer floppy disk
Time in Seconds
100
Compress floppy disk
Transfer VM overlay
80
60
40
20
0
AbiWord
© 2009-2010 M. Satyanarayanan
GIMP
Gnumeric
Kpresenter
PathFind
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
SnapFind
Null
27
When Bandwidth Drops to 10Mbps
Other
350
Resume VM
Largest standard deviation is 3.4% of mean
300
Apply VM overlay
Decompress VM overlay
Time in Seconds
250
Transfer floppy disk
Compress floppy disk
200
Transfer VM overlay
150
100
50
0
AbiWord
© 2009-2010 M. Satyanarayanan
GIMP
Gnumeric
Kpresenter
PathFind
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
SnapFind
Null
28
In Closing 
Leverage the Cloud!
(but keep the Swiss Army Knife handy for emergencies)
© 2009-2010 M. Satyanarayanan
DAC Workshop on Mobile and Cloud Computing, Anaheim, 2010-06-14
29