HUGEPAGE SUPPORT IN LINUX
BASED VIRTUALIZATION
 Group members–




Aditya Gadre
Keshav Darak
Kaustubh Kabra
Ashwin Vasani
 External guide–
Mr. Kedar Sovani
DREAMZ Group.
B3054204
B3054243
B3054279
B3054365
Internal guideProf Mukta Takalikar
PICT,Pune.
1
PROBLEM DEFINITION
To design a memory management unit
(MMU) based paging mechanism in Xen
hypervisor which will add support for huge
pages (2MB) such that 4 KB pages and huge
pages will now co-exist for a paravirtualized
domain kernel to improve performance.
2
ARCHITECTURE
REVIEW
APPLICATIONS &
FUTURE
ENHANCEMENTS
X-XEN
BUSINESS
VALUE
CURRENT
SCENARIO
CHALLENGES
IMPLEMENTATION
PERFORMANCE
3
HUGE PAGE
A.Huge page translation reduces the normal
page walk by 1 level.
B. The Huge page TLB cache improves the performance by
increasing the speed by15% to 20%.
4
Virtual address
PGD
PMD
PTE
Page Offset
3
Only 2 memory
4 KB page
access additional
Singleton
to reach to a byte
1
2
PS
10
3 memory
acccess
2 MB page
additional to
Huge Page
reach to a byte
PGD
PTE
Physical
address
Physical
address
Page Offset
Virtual address
5
HUGE PAGE
A. Huge page translation reduces the normal page walk by 1
level.
B.The Huge page TLB cache improves the
performance by increasing the speed
by15% to 20%.
6
TLB Scenario
RAM
TLB Cache
1
entry
2 MB
512 dispersed
4 KB pages
VA
PA
512
entries
7
Bare Metal
Hardware
VS
XEN
Hardware
8
XEN ??
9
ARCHITECTURE
REVIEW
APPLICATIONS &
FUTURE
ENHANCEMENTS
X-XEN
BUSINESS
VALUE
CURRENT
SCENARIO
CHALLENGES
IMPLEMENTATION
PERFORMANCE
10
11
12
13
ARCHITECTURE
REVIEW
APPLICATIONS &
FUTURE
ENHANCEMENTS
X-XEN
BUSINESS
VALUE
CURRENT
SCENARIO
CHALLENGES
IMPLEMENTATION
PERFORMANCE
14
Requesting
2MB
Requesting
4KB
DOMU
(1)
DOM U
(2)
Memory
Hypervisor
Request
denied
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
RAM
15
16
PROBLEM DEFINITION
To design a memory management unit
(MMU) based paging mechanism in Xen
hypervisor which will add support for huge
pages (2MB) such that 4 KB pages and huge
pages will now co-exist for a paravirtualized
domain kernel to improve performance.
17
ARCHITECTURE
REVIEW
APPLICATIONS &
FUTURE
ENHANCEMENTS
X-XEN
BUSINESS
VALUE
CURRENT
SCENARIO
CHALLENGES
IMPLEMENTATION
PERFORMANCE
18
CONCEPTS
Kernel
Domain Kernel
3
4
6
1
5
2
VA
1
Domain Kernel
3
4
VA_to_PFN
1
2
3
4
2
6
5
5
6
VA_to_PFN
5
PFN
6
1
2
3
4
RAM
Hypervisor
P2M
3
2
4
1
1
4
5
•VA : Virtual Address in Kernel
•PFN : Physical Frame Number
•MFN : Machine Frame Number
•P2M Mapping: Physical to Machine Mapping
6
1
6
5
3
MFN
19
IMPLEMENTATION OVERVIEW
Requesting
2MB
4KB
DOMU
(1)
Hypervisor
Boot time
reservation
of memory
4KB
4KB
4KB
4KB
2MB 4KB2MB4KB
4KB
2MB
4KB
2MB4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
4KB
RAM
4KB
Implementation steps :
1. Domain creation
2. Domain booting
3. Demand supply
Implementation steps :
1. Domain creation
2. Domain booting
3. Demand supply
22
Domain’s Config
Domain creation
struct domain{
memory=900
----------------------
hugepage_num;
hugepage_list;
hugepage_num=50
50
-------------------------------------------
Domain Create
MFNs
};
Xen’s Allocator
RAM
Xen
MFNs
23
Implementation steps :
1. Domain creation
2. Domain booting
3. Demand supply
24
Domain booting
xen_extra_mem_start
Kernel
800M
900M
PFNs
Kernel Booting
Hypercall to
Get hugepage count
Xen Hypervisor
struct domain{
hugepage_num;
hugepage_list;
};
50
25
Implementation steps :
1. Domain creation
2. Domain booting
3. Demand supply
26
Demand
xen_extra_mem_start
Kernel
800M
Hugepage
Request
PFNs
P
F
N
Kernel
Hypercall to
Allocate hugepage
900M
MFN
P2M
Xen Hypervisor
MFN
RAM
Xen
struct domain{
hugepage_num;
hugepage_list;
};
MFNs
27
FINAL VISION(Transparent Huge Pages)
Requesting
4KB
Requesting
2MB
DOMU
(1)
DOM U
(2)
Hypervisor
4
K
4
K
4
2MB
K
2MB
2MB
4
K
4
K
2MB
2MB
2MB
2MB
2MB
2MB
2MB
2MB
2MB
RAM
28
ARCHITECTURE
REVIEW
APPLICATIONS
& FUTURE
ENHANCEMENT
S
X-XEN
BUSINESS
VALUE
CURRENT
SCENARIOAMAZON
DEMO
CHALLENGES
IMPLEMENTATION
PERFORMANCE
29
PERFORMANCE GRAPHS
Threads =4
Threads =8
Threads =6
Threads =10
X axis: Memory read and wrote (in MBs)
Y axis: Time required (in seconds)
30
ARCHITECTURE
REVIEW
APPLICATIONS &
FUTURE
ENHANCEMENTS
X-XEN
BUSINESS
VALUE
CURRENT
SCENARIO
CHALLENGES
IMPLEMENTATION
PERFORMANCE
31
Challenge 1 :
Paging
DOCUMENTATION
Challenge 2 : Handling read-only ptes in guest domains
Read-Only Access to
Guest Page Tables
1
2
Read-Only Access to
Guest Page Tables
1
3
2
3
Xen Hypervisor
2
1
2
3
1
3
33
Challenge 3 (1/2): Aligned PFNs and unaligned MFNs
Domain Kernel
512 aligned pfns for
Huge Pages
1
2
3
4
7
8
PFN
2MB Boundary
Xen Hypervisor
Unaligned non- contiguous 512MFNs
2
1
11
1
3
4
8
7
MFN
34
Challenge 3 (1/2): Aligned PFNs and unaligned MFNs
Domain Kernel
Unaligned un-contiguous PFN
1
2
5
8
5
7
PFN
Xen Hypervisor
Aligned (512 )MFN
2
1
11
1
2MB Boundary
5
8
9
MFN
35
Our solution: Aligned PFNs and Aligned MFNs
Domain Kernel
512 aligned pfns for
Huge Pages
1
2
3
4
7
8
PFN
2MB Boundary
Hypervisor can complete request successfully
Aligned (512 )MFN
2
1
11
1
2MB Boundary
2
8
9
MFN
36
ARCHITECTURE
REVIEW
APPLICATIONS &
FUTURE
ENHANCEMENTS
X-XEN
BUSINESS
VALUE
CURRENT
SCENARIO
CHALLENGES
IMPLEMENTATION
PERFORMANCE
37
APPLICATIONS
38
FUTURE ENHANCEMENTS
• Existing Live Migration implementations
need to be compatible with this
implementation.
• Support for 1GB huge pages.
39
REFERENCES
• http://linuxgazette.net/155/krishnakumar.html : HugeTLB Large Page Support in the Linux Kernel
• “Memory Management in Linux” by Abhishek Nayani, Mel
Gorman & Rodrigo
• “XEN Memory Management (Intel IA-32)” by Andr´es Krapf.
• “Memory Overcommit without the commitment” by Dan
Magenheimer, Oracle Corp. Xen Summit, June 2008
• “Xen and art of virtualization” By P. Barham, B.Dragovic,
K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt,
40
and A. Warfield. In Proc. SOSP, 2003
CONTRIBUTION
 Patch submitted for review to Jeremy Fitizhardinge, Keir
Fraser and Xen-Devel community
 http://tinyurl.com/HugePagea2k2Patch1
 www.github.com/a2k2/
 Kernel-PV-hugepages
 Xen-hugepages
 XEN memory: Enlightening memory management in
XEN - Team a2k2
 www.ashwinvasani.wordpress.com
 Summarized documentation of the project
 www.xxen-docs.co.cc
41
AWARDS
42
ENLIGHTENERS
• Kedar Sovani
(Marvell & Former
Maintainer of FedoraARM)
• Prof. Mukta Takalikar
(PICT)
• Jeremy Fitizhardinge
(PV-Kernel maintainer)
• Keir Fraser
(Xen maintainer)
• Dan Magenheimer
(Tmem maintainer)
• Tim Deegan
(Citrix)
43
THANK YOU.
?
44