HPC-VMs: Virtual Machines in High
Performance Computing Systems
Albert Reuther
IEEE-HPEC 2012
September 11, 2012
This work is sponsored by the Department of the Air Force under Air Force contract FA8721-05-C-0002.
Opinions, interpretations, conclusions and recommendations are those of the author and are not
necessarily endorsed by the United States Government.
Outline
• Introduction to Virtual Machines
• VM Features
• VMs in HPC
• Launch Time Results
• Summary and Future Work
HPC VMs- 2
AIR 9/11/12
Operating System Basics
User(s)
User Interfaces
Applications
APIs
Libraries/
Frameworks
OS Kernel &
Device Drivers
Hardware
• Applications and their User Interfaces – Programs
that users run
• Application Programming Interface (API) – Rules
and specifications for libraries and frameworks
• Libraries/Frameworks – Reusable software
routines for building applications
• Operating System (OS) Kernel – Manager of
computer hardware resources and of common
services for application software
• Hardware – Physical components of the computer
• Manages and controls shared hardware resources
• Provides common services to applications
• Abstracts hardware for users and applications
HPC VMs- 3
AIR 9/11/12
Silbershatz, Galvin and Gagne, Operating System Concepts, Addison Wesley, 2011.
Virtual Machines
User(s)
User Interfaces
Applications
APIs
Libraries/
Frameworks
Operating System
Virtual Machine (VM)
• Operating System (OS) – Manager of computer hardware
resources and of common services for application
software
• Virtual Machine (VM) – Software implementation of a
computer that executes programs like a computer
machine
• Hypervisor – Virtual operating platform that monitors the
execution of one or more VMs
• Hardware – Physical components of the computer
• of the computer
Virtual Machines
Hypervisor
HPC VMs- 4
AIR 9/11/12
Dittner and Rule, Best Damn Server Virtualization Book Period, Syngress, 2007.
OS
VM
OS
VM
OS
VM
Hypervisor
Mon
•
systems
Allocate hardware resources
App
Mon
• Emulate the underlying hardware
• Isolate and encapsulate guest operating
App
Mon
Hardware
App
Mgmt
Cloud Computing Service Paradigms
Provided
by User,
Admin,
etc.
Infrastructure as a
Service (IaaS)
Platform as a
Service (PaaS)
Software as a
Service (SaaS)
User(s)
User(s)
User(s)
User Interfaces
Applications
APIs
Libraries/
Frameworks
User Interfaces
Applications
APIs
Libraries/
Frameworks
User Interfaces
Applications
APIs
Libraries/
Frameworks
Operating System
Operating System
Virtual Machine (VM)
Virtual Machine (VM)
Hypervisor
Hypervisor
Hypervisor
Hardware
Hardware
Hardware
Operating System
Virtual Machine (VM)
Provided
as a
Service
Provide hardware to
execute VMs
Provide software
execution platform
Provide entire
software application
Each service paradigm has different security
and reliability implications
HPC VMs- 5
AIR 9/11/12
Cloud Computing Service Paradigms
Provided
by User,
Admin,
etc.
Infrastructure as a
Service (IaaS)
Platform as a
Service (PaaS)
Software as a
Service (SaaS)
User(s)
User(s)
User(s)
User Interfaces
Applications
APIs
Libraries/
Frameworks
User Interfaces
Applications
APIs
Libraries/
Frameworks
User Interfaces
Applications
APIs
Libraries/
Frameworks
Operating System
Operating System
Virtual Machine (VM)
Virtual Machine (VM)
Hypervisor
Hypervisor
Hypervisor
Hardware
Hardware
Hardware
Operating System
Virtual Machine (VM)
Provided
as a
Service
Utility
Cloud
Examples
HPC VMs- 6
AIR 9/11/12
Amazon Elastic Compute
Cloud (EC2), ITricity, Joyent,
Rackspace, VMWare Cloud
Amazon Web Services (AWS),
Amazon Simple Storage
Service (S3), Windows Azure,
FaceBook Apps
SalesForce.com, Google
Apps, Gmail, Microsoft Suite
365, NetFlix Streaming,
FaceBook
A Brief History of Virtual Machines
Servers
Mainframes
with
Terminals
Distributed
Servers with
Distributed PCs
IBM
CP/CMP
(first VMs)
DEC VAX
(virtual
memory)
Virtualization
Multics
OSs
LANs
Storage
Unix
System V
ARPANET
First
Message
Directattached
Tapes
BSD
1.0
Ethernet
First
Deployment
Directattached
Disks
Portable,
Heterogeneous,
Ubiquitous VMs
Linux 1.0
AIX
Kernel
Solaris
HP-UX Windows Windows
3.1
NT
Ethernet
10BASE5
Linux 2.6
Kernel
Windows
XP
Ethernet
100BASE-T
Ethernet
10BASE-T
Disk
Farms
Web Services
with Browser
(PCs, Laptops,
Mobile Devices)
Software
VMs
(Java VM)
Hardware
Emulation
DEC
VMS
Central
Servers with
Distributed
PCs
Redundant
Array of
Independent
Disks (RAID)
Ethernet
10GBASE-T
Ethernet
1000BASE-T
Storage- NetworkAttached Attached
Networks Storage
(SANs)
(NASs)
Distributed
Replicated,
Network
Storage
1960s
1970s
1980s
1990s
2000s
2010s
• Distributed computing has been in CS research since Multics and ARPANET.
• Cloud Computing is the convergence and commercialization of many distributed
computing areas.
Mainframe Era
LAN/Web Era
Color
HPC VMs- 7
AIR 9/11/12
Legend:
PC Era
Cloud Era
Types of Modern Virtual Machines
Ring 3
• Emulation: Translate all instructions from guest
OS to host hardware
• Paravirtualization: Partial simulation of underlying
hardware for guest OSs
Ring 2
Ring 1
Ring 0
Kernel
– Jump table for protected guest OS instructions
– Some guest OS modifications
Device Drivers
• Full Virtualization: Complete simulation of
underlying hardware for guest OSs
Device Drivers
User
– Hardware support for virtualization
Virtual Machines
Virtual Machines
HPC VMs- 8
AIR 9/11/12
Mgmt
App
App
OS
VM
OS
VM
OS
VM
Hypervisor
Mon
Hypervisor
• Type 2: Host OS
and hypervisor
App
Mon
OS
VM
• Type 1: Bare metal
hypervisor
Mon
OS
VM
Mon
OS
VM
Mon
App
Mon
App
App
Mgmt
Host Operating System
OS
Protection
Rings
Outline
• Introduction to Virtual Machines
• VM Features
• VMs in HPC
• Launch Time Results
• Summary and Future Work
HPC VMs- 9
AIR 9/11/12
VM Resources
• Choose number of CPU
cores
• Choose main memory size
• Choose devices to present to
guest OS
Virtual Machines
App
App
OS
VM
OS
VM
OS
VM
Hypervisor
Hardware
HPC VMs- 10
AIR 9/11/12
Mon
App
Mon
Network
Storage
Display
Audio
USB
Etc.
Mon
–
–
–
–
–
–
Mgmt
Common Virtual Machines
Name
Organization
/License
Type
KVM
Qumranet
1
Virtualization
Up to near
native
libvert
Microsoft Virtual
Server
Microsoft
2
Virtualization
Up to near
native
COM
Microsoft
Hyper-V
Microsoft
1
Virtualization
Up to near
native
libvert, WMI
QEMU
Fabrice Bellard
and others
2
Emulation
Less than
native
libvirt, libguestfs
Virtual Box
Oracle
2
Virtualization
Up to near
native
Command line,
libvirt, Main API
VMware
Workstation
and Server
VMware (EMC)
2
Paravirtualization and
Virtualization
Up to near
native
libvirt, VMware
VMware ESX
VMware (EMC)
1
Virtualization
Up to near
native
libvirt, VMware
Xen
Xensource and
Citrix
1
Paravirtualization and
Hardware Virtualization
Up to native
libvirt, xenaccess,
XenAPI
HPC VMs- 11
AIR 9/11/12
Virtualization
Speed
From: http://en.wikipedia.org/wiki/Comparison_of_platform_virtual_machines
API
Enterprise VM Trade-Offs
Advantages
Challenges
• Server consolidation with
isolation
• Performance
• Rapid provisioning (no
hardware purchase needed)
• VM migration between
hardware servers
• Flexibility of running multiple
OS types
• Flexible virtual hardware
configuration
• Probably better security
stance
HPC VMs- 12
AIR 9/11/12
– Network
– I/O (disks, etc.)
• Management of VM images
• Load balancing on hardware
But HPC is Different.
It is all about
performance and
productivity.
Outline
• Introduction to Virtual Machines
• VM Features
• VMs in HPC
• Launch Time Results
• Summary and Future Work
HPC VMs- 13
AIR 9/11/12
HPC and Virtual Machines
Two Common Approaches
• HPC on an IaaS cloud
– High cost for consistent use
– Requires good bandwidth to
cloud provider
– Security and privacy
considerations
HPC VMs- 14
AIR 9/11/12
• Static VMs on hardware with
scheduler
– Every user incurs VM
performance penalties
– Static mix of VM configurations
VMs on LLGrid:
Providing Productivity to Users
Service Nodes
Interactive
Users
Compute Nodes
Cluster Switch
Network
Storage
Resource Manager
Login Node(s)
Configuration
Server
LLAN
To Lincoln LAN
• Two primary HPC VM users
– Older or different OS requirement
LAN Switch
• Scheduler executes VMs on cluster as
OS encapsulation environment
 Code validated on legacy OS (e.g. RHEL 3)
– Deploy on Type 2 hypervisor
 Application only available on Solaris
– Hypervisor API enables VM configuration
– Prototyping heterogeneous distributed
computing environments
– Intercept VM shutdown for job shutdown
– Provide set of “standard” VMs
– Store user VMs in user account
HPC VMs- 15
AIR 9/11/12
VMs on LLGrid Features
• Stripped virtual machines
– Stock kernel
– Scientific libraries
– No services or applications
• Job execution written into /etc/init.d script
• Job completion triggers VM shutdown/teardown
• Central file system shared through host OS
• Single job slots and job array launching
• Jobslot overloading available
– Launch more than one VM per jobslot
– Constrained by compute node resources (memory, cores, etc.)
HPC VMs- 16
AIR 9/11/12
Outline
• Introduction to Virtual Machines
• VM Features
• VMs in HPC
• Launch Time Results
• Summary and Future Work
HPC VMs- 17
AIR 9/11/12
Launch Time Experiment Setup
Service Nodes
Compute Nodes
Cluster Switch
Network
Storage
Resource Manager
Login Node(s)
Configuration
Server
LAN Switch
•
•
•
•
•
•
•
Dell PowerEdge 1955 blades
Dual-dual core 3.2 GHz Xeon CPU
8 GB RAM per blade
10GigE core network, 1GigE to blades
DDN SFA 10K storage array
Grid Engine ver. 6.2u5 scheduler
VM images: Debian Linux 6.0.4 i386
HPC VMs- 18
AIR 9/11/12
• Eight dedicated nodes
• Compare un-optimized VMs with
optimized VM images
• Varied jobslots launched and jobslot
overloads
• Socket-based time logger
Launch Time Results
• Optimized VMs launch significantly faster than unoptimized
• Overloading jobslots does not impact launch time much
HPC VMs- 19
AIR 9/11/12
Outline
• Introduction to Virtual Machines
• VM Features
• VMs in HPC
• Launch Time Results
• Summary and Future Work
HPC VMs- 20
AIR 9/11/12
Summary and Future Work
• VMs at the heart of Cloud Computing
• Can judiciously use VMs in HPC environments
– Encapsulate older OS environments rather than re-validate software
– Enable large-scale distributed environments in clusters
• VM launches add modest overhead to job launches
• Can add performance penalty to disk and network I/O
• Future work
–
–
–
–
HPC VMs- 21
AIR 9/11/12
Parallel job launches (MPI, PGAS ,etc.)
Support VMWare virtual machines
Demonstrate persistent services in VMs (including dynamic DNS)
Transition technology to researchers and sponsored projects