Mobile I/O virtualization
Harvey Tuch, Staff Engineer, Mobile Virtualization Platform
January 25th 2012
Sponsored by MIT and VMware Academic Programs
VMware: www.vmware.com
VMware Labs: labs.vmware.com
© 2010 VMware Inc. All rights reserved
Agenda
 Smartphone hardware
 I/O virtualization techniques
 Examples from MVP hypervisor
• Storage virtualization
• GPS virtualization
• Telephony virtualization
2
Smartphone hardware
3
Computer System Organization
CPU
Memory
MMU
Controller
Local Bus
Interface
High-Speed
I/O Bus
NIC
Controller
Bridge
Frame
Buffer
LA
N
Low-Speed
CD-ROM
4
USB
I/O Bus
Evolution of mobile phone
to Mobile Personal Computer
5
6
Recent smartphone specs sample (source: pdadb.net)
6
Copyright ® VMware, Inc. All Rights Reserved.
Hardware trends
 Example smartphone circa 2012
• > 1GHz dual or quad core ARM Cortex-A9/A15 or Snapdragon
• 1GB SDRAM
• 512MB internal NAND Flash
• 32GB microSD
• Integrated GPS, > 10MP camera, 3G/4G (LTE), 802.11, Bluetooth
• 480x800 OLED
• Ice Cream Sandwich Android OS
 Tablets, smartbooks, etc.
7
Copyright ® VMware, Inc. All Rights Reserved.
Smartphone physical hardware
Smartphone internals
• Processor core – ARM
System-on-Chip (SoC)
• Power efficiency, battery life
Interrupt
controller
Timers
DMA engine
LCD
controller
• E.g. 8 DMIPS/mW (Cortex-A9)
GPU
Camera
controller
USB
controller
• ARMv7 ISA
DSP
SD card
controller
GPIO
Power
management
UART
Memory
controller
ARM core
(e.g. Cortex A-8)
Cache
MMU
• Devices – highly integrated
• SoC organization
SDRAM
NAND Flash
GSM modem
GPS
WiFi
Bluetooth
LCD/touchscreen
Microphone/speaker
Camera
Accelerometer/compass
Antenna
Battery
• Plethora of devices (IP blocks)
• Limited standardization
8
• Storage – solid state NAND
Agenda
 Smartphone hardware
 I/O virtualization techniques
 Examples from MVP hypervisor
• Storage virtualization
• GPS virtualization
• Telephony virtualization
9
I/O virtualization techniques
10
Device virtualization goals
 Present to guest a “virtual” platform (set of devices)
• Guest applications, middleware, OS expect standard set of mobile devices
• E.g. touchscreen, display, storage, GPS, Bluetooth, WiFi, GSM, GPU,
cameras, accelerometers, audio, keyboard, etc.
 Multiplex physical hardware between VMs
• Securely sharing physical hardware across domains
• Efficiency
• Fidelity
• Portability
• Interposition (checkpointing, live migration)
• Minimizing engineering effort (cost)
11
Device virtualization techniques
 Device emulation
• VMM emulates registers and memory map of physical HW devices
 Paravirtualization
• VMM provides optimized high-level API for guest devices
 Pass-thru
• VMM provides guest direct access to physical HW devices
12
Device virtualization techniques – device emulation
 VMM emulates registers and memory map of physical HW devices
 Leverage existing guest OS driver support
 Interposable
✗Emulation overhead
✗VMM implementation cost
Guest Operating System
ACME UART driver
Write FIFO ‘a’
Read status register CSR
Write FIFO ‘b’
…..
ACME UART device model
Physical FOOBAR UART
Hypervisor
Write FIFO ‘a’
Write FIFO ‘b’
Read status register FSR
13
…..
Device virtualization techniques - paravirtualization
 VMM provides optimized high-level API for guest devices
 Higher performance possible
 Simplified VMM
 Interposable
✗Requires custom drivers/libraries per guest
Guest Operating System
PV UART driver
Write shared buffer
‘abcdefghijklmno…’
…..
PV UART device model
Physical FOOBAR UART
Hypervisor
Write FIFO ‘a’
Write FIFO ‘b’
Read status register FSR
14
…..
Device virtualization techniques – pass-thru
 VMM provides guest direct access to physical HW devices
✓Highest performance, even simpler VMM
✗I/O MMU required for safety (DMA danger)
✗Multiplexing HW devices requires extra HW support
✗Interposition hard (hybrid models separate control/data paths)
Guest Operating System
Write FIFO ‘a’
ACME UART driver
Read status register
Write FIFO ‘b’
…..
Physical ACME UART
Hypervisor
15
Bare-metal (Type-1)
hypervisor
architecture
Bare-metal
architecture
(type-1)
Linux
Linux
XP
Vista
Virtual Machine Monitor
Hardware
directly on
physical HW
! Executes
 Execute directly
on physical
HW
 Preferred architecture
for high-end
serversservers
for high-end
! Preferred architecture
 E.g. VMware!ESX,
Xen, Microsoft
Hyper-V
E.g. VMware
ESX, Xen, Microsoft
Hyper-V
16
MacOS
Hosted (Type-2)
hypervisor
architecture
Hosted
architecture
(type-2)
Guest OS (Linux)
User App
World
Kernel
Module
Host OS (Window XP)
Virtual Machine Monitor
Hardware
host OS,
Windows,
OS XWindows
! Executes alongside
 Executes alongside
existingexisting
host OS,
e.g.e.g.
Mac
OS X,Linux,
Linux,
host
device drivers
and resourcemanagement
management
! Leverage
 Leverage host
device
drivers
and resources
VMware Workstation, KVM,
Parallels
Desktop
! E.g.
 E.g. VMware
Workstation/Fusion,
KVM,
Parallels
Desktop
17
9
Copyright ® VMware, Inc. All Rights Reserved.
1
VMware MVP system architecture
 Personal applications run
natively
 Corporate applications run in a
virtual machine
18
Copyright ® VMware, Inc. All Rights Reserved.
1
MVP as a hosted (Type-2) hypervisor
 Hosted hypervisor
benefits:
• Tackle SoC diversity
• Reduced TTM
• Zero virtualization overhead
for personal phone.
• Almost all components can
be downloaded OTA (app
store).
19
Copyright ® VMware, Inc. All Rights Reserved.
Platform
virtualization
Device
virtualization
stack
Guest OS (Linux)
User App
Device I/ O
Network,
Disk,
Display,
Kernel
Module
Host OS (Window XP)
Virtual Machine Monitor
Hardware
 Frontend
(RHS)smartphone devices/subsystems
!Virtualized
• Guest
OS – VMM storage,
interfacetelephony,
for virtual devices
GPS, frame buffer, touchscreen, battery, audio
• Networking,
used extensively
 Backend
(LHS)
!Paravirtualization
• Hypervisor
implementation
of virtual
device
semantics
onAndroid
host middleware)
levels of the stack
(HW,
kernel
subsytem,
• Different
20
32
Copyright ® VMware, Inc. All Rights Reserved.
Frontend virtualization – where to slice?
Application
R: 137
G: 203
B: 223
Application level PV
R: 192
G: 192
B: 192
Libraries, middleware
Library level PV
Operating System
R: 248
G: 152
B: 29
Device level PV
Device emulation
Virtualization Layer
21
R: 217
G: 84
B: 30
Backend implementation
 Host user space (application level)
• Translate guest device I/Os to standard kernel/library calls (e.g. POSIX)
• E.g. write ‘a’ to /dev/ttyS0 for UART transmit
✓Portable, simple, language independent
✗System call overhead, latency
 Host kernel
• Translate guest device I/Os to kernel level function calls
✓Lowest overhead, direct access to device subsystem
✗Kernel dependent, modularity issues, increased TCB
 Hardware (pass-thru)
22
Agenda
 Smartphone hardware
 I/O virtualization techniques
 Examples from MVP hypervisor
• Storage virtualization
• GPS virtualization
• Telephony virtualization
23
Storage virtualization
24
Smartphone storage devices
 Internal storage
• NAND flash devices
• Software Flash Translation Layer (FTL)
• Limited size (256MB – several GB)
• Kernel, application code, libraries, middleware
 External storage
• microSD cards
• Hardware FTL
• Up to 32GB today (2TB future)
• Economics of semiconductor scaling
• Optimized for cost, media workloads
• Application data (and some code)
25
Storagevirtualization
virtualizationstack
Storage
V
M
X
(
S
to
r
a
g
e
th
r
e
a
d
,
g
u
e
s
tim
a
g
e
fo
r
m
a
t
p
r
o
c
e
s
s
in
g
,
e
n
c
r
y
p
tio
n
)
H
o
s
t"
u
n
ix
"
u
s
e
rs
p
a
c
e
H
o
s
tL
in
u
x
K
e
r
n
e
l
V
F
A
T
G
u
e
s
t
A
p
p
lic
a
tio
n
H
o
s
tlib
c
V
F
S
lib
c
M
k
s
c
k
+
S
h
a
r
e
d
M
e
m
e
x
t3
B
lo
c
k
L
a
y
e
r
e
x
t3
N
A
N
D
F
la
s
h
F
ile
s
y
s
te
m
N
A
N
D
L
a
y
e
r
M
M
C
/S
D
D
r
iv
e
r
N
A
N
D
C
h
ip
D
r
iv
e
r
m
ic
r
o
S
D
C
a
r
d
N
A
N
D
C
H
IP
B
lo
c
k
L
a
y
e
r
P
V
B
lo
c
k
D
r
iv
e
r
F
T
L
M
M
C
/S
D
L
a
y
e
r
V
F
S
P
h
y
s
ic
a
l
H
a
r
d
w
a
r
e
V
ir
t
u
a
l
H
a
r
d
w
a
r
e
F
T
L
N
A
N
D
26
34
Copyright ® VMware, Inc. All Rights Reserved.
G
u
e
s
t"
u
n
ix
"
u
s
e
rs
p
a
c
e
G
u
e
s
tL
in
u
x
K
e
r
n
e
l
VM image storage on SD cards
 Why?
• Storage footprint of guest may be several GB (including checkpoint images)
 Challenges
• Performance
• Reliability
• Security
27
VM image storage on SD cards - performance
 FTL optimized for cost, media workloads
• Poor non-sequential I/O performance
• VM workloads exhibit far more non-sequential I/O than MP3 streaming,
photos, videos, etc.
28
VM image storage on SD cards - performance
.
29
VM image storage on SD cards - performance
.
30
VM image storage on SD cards - performance
.
31
VM image storage on SD cards – other challenges
 Reliability
• Dropped phone, whoops
• Battery dies
• Host software crashes (in particular at kernel level)
 Security
• SD cards are FAT formatted for compatibility
• No access controls
• Malicious host apps can read/write VM images
32
Log structured filesystems (LSF)
 An excursion back in time to… 1991…
• Slow seek latency on mechnical disks, CPUs getting faster
• Non-sequential file writes expensive
• Solution: every file update gets appended to a sequential “log”
• Maintain meta-data to later locate the most recent block for a file in the log
• Reads mostly hit in the page cache, don’t worry about non-sequentiality
• Mendel Rosenblum and John K. Ousterhout. The design and implementation
of a log-structured file system. In ACM Symposium on Operating System
Principles, 1991.
 Benefits beyond performance…
• Meta-data only updated after new data is on disk
• Old data still exists earlier in log
• Crash recovery discovers latest valid block on disk
33
LSF example
Log
3
Latest contents for block
Old contents for block
Logical block
0
1
2
3
4
...
34
Log location
0
...
LSF example
Log
3
0
Latest contents for block
Old contents for block
Logical block
0
1
2
3
4
...
35
Log location
1
0
...
LSF example
Log
3
0
1
Latest contents for block
Old contents for block
Logical block
0
1
2
3
4
...
36
Log location
1
2
0
...
LSF example
Log
3
0
1
3
Latest contents for block
Old contents for block
Logical block
0
1
2
3
4
...
37
Log location
1
2
3
...
Storage backend - Logging block store (LBS)
 Backend represents VM disk & checkpoint images with files on
host
• Similar format to a LFS (performance, reliability)
• Split data and meta-data between internal & external storage
• Encryption & integrity checks on data file
• Meta-data and encryption keys located on protected internal storage (security)
38
LBS performance
.
39
Agenda
 Smartphone hardware
 I/O virtualization techniques
 Examples from MVP hypervisor
• Storage virtualization
• GPS virtualization
• Telephony virtualization
40
GPS virtualization
41
GPS virtualization
GPS virtualization
Guest
Host
MVPLocationServer
(Java)
Guest application
(Java)
Android Java Location
libraries
Android Java Location
libraries
libgps_physhw.so
(JNI)
TCP loopback over
libgps_mvp.so (JNI)
virtualized network
Host kernel GPS
device driver
GPS physical
hardware
42
33
Copyright ® VMware, Inc. All Rights Reserved.
Agenda
 Smartphone hardware
 I/O virtualization techniques
 Examples from MVP hypervisor
• Storage virtualization
• GPS virtualization
• Telephony virtualization
43
Telephony virtualization
44
Telephony virtualization
 GSM
• Dual SIM smartphones (geography dependent)
• Multiple International Mobile Subscriber Identities (IMSIs) per SIM (geo dep)
• GSM Alternate Line Service (ALS) (network dependent)
 Network
• Call forwarding, ala Google Voice, Line2
 Voice-over-IP (VoIP)
• Realistic with 4G/LTE
• Integration with enterprise Unified Commmunications (UC)
45