Abstract Image Management and
Universal Image Registration for
Cloud and HPC Infrastructures
Javier Diaz, Gregor von Laszewski,
Fugang Wang and Geoffrey Fox
Community Grids Lab
Pervasive Technology Institute
https://portal.futuregrid.org
Indiana
University
Motivation
• FutureGrid (FG) is a testbed providing users with
grid, cloud, and high performance computing
resources
• One of the goals of FutureGrid is to provide a
testbed to perform experiments in a reproducible
way among different infrastructures
• We need mechanism to ease the use of these
infrastructures
• FG Image Management framework allows users to
easily create customized environments by placing
suitable images onto the FG resources
https://portal.futuregrid.org
Introduction I
• Image management is a key component in any modern
compute infrastructure (virtualized or non-virtualized)
• Processes part of the image management life-cycle:
Creating and
Customizing
Images
User selects
properties
and software stack
features
meeting his/her
requirements
(b)
Storing
Images
Abstract
Image
Repository
(c)
Registering
Images
Adapting the Images
(a)
http://futuregrid.org
(d)
Instantiating
Images
Nimbus
Eucalyptus
OpenStack
OpenNebula
Bare Metal
Introduction II
• Targeting multiple infrastructures amplifies the
need for mechanisms to ease these image
management processes
• We have identified two mechanisms
– Introduce standards and best practices to interface with
the infrastructure (OVF, OCCI, Amazon EC2)
– Provide tools that interface with these standards and
expose the functionality to the users while hiding the
underlying complexities
• Otherwise, only the most experienced users will be able
to manage images for multiple infrastructures under great
investment of time
https://portal.futuregrid.org
FutureGrid Image Management
Framework
• Framework provides users with the tools needed to
ease image management across infrastructures
• Users choose the software stacks of their images and
the infrastructure/s
• Targets end-to-end workflow of the image life-cycle
• Create, store, register and deploy images for both
virtualized and non-virtualized resources in a
transparent way
• Allows users to have access to bare-metal
provisioning (departure from typical HPC centers)
– Users are not locked into a specific computational
environment offered typically by HPC centers
https://portal.futuregrid.org
Architectural Overview
Image
Management
Client
Portal
FG Shell
Image
Management
Server
Image
Generation
Image
Repository
API
Image
Registration
Image
Instantiation
External Services:
Chef, Security tools
IaaS and Bare-Metal HPC
Infrastructures
Cloud IaaS
Frameworks
Nimbus
Eucalyptus
AWS
OpenNebula
OpenStack
HPC Clusters
Bare Metal
https://portal.futuregrid.org
Image Generation
• Creates images according to
user’s specifications:
Command Line Tools
Requirements:
OS, version, hadrware,...
• OS type and version
• Architecture
• Software Packages
Yes
• Software installation may be
aided by Chef
Retrieve
Image from
Repository
• Images are not aimed to any
specific infrastructure
• Image stored in Repository or
returned to user
https://portal.futuregrid.org
Matching Base
Image in the
Repository?
No
Generate Image
Image Gen. Server
OpenNebula
Base OS
VM
VM
CentOS 5 VM
CentOS 6
Ubuntu 12 X86_64
X86_64
X86
Base Software
Base Image
FG Software
Install Software
Cloud Software
Update Image
User Software
User's Image
Store in Image
Repository
Image Repository
• Service to query, store, and update images
• Unique interface to store various kind of images for
different systems
• Images are augmented with some metadata which is
maintained in a searchable catalog
• Keep data related with the usage to assist performance
monitoring and accounting
• Independent from the storage back-end. It supports a
variety of them and new plugins can be easily created
https://portal.futuregrid.org
Image Metadata
User Metadata
imgId
Image’s unique identifier
Field
Name
owner
owner
userId
os
Operating system
User’s unique
identifier
description
Description of the image
fsCap
Disk max usage (quota)
tag
Image’s keywords
fsUsed
Disk space used
vmType
Virtual machine type
lastLogin
Last time user used
the framework
imgType
Aim of the image
status
permission
Access permission
Active, pending,
disable
imgStatus
Status of the image
role
Admin, User
createdDate
Upload date
ownedimg
# of owned images
lastAccess
Last time the image was accessed
Field Name
Description
accessCount # times the image has been
accessed
size
Size of the image
Description
Image Registration I
• Adapts and registers images into specific
infrastructures
• Two main infrastructures types are considered
to adapt the image:
– HPC: Create network bootable images that can
run in bare-metal machines (xCAT/Moab)
– Cloud: Convert the images in VM disks and
enable VM’s contextualization for the selected
cloud
https://portal.futuregrid.org
Image Registration II
• User specifies where to
Command Line Tools
register the image
Requirements: Image,
• Optionally, user can select Kernel, Infrastructure
User's Image
kernel from a catalog
Customize Image for:
• Decides if an image is
HPC
Eucalyptus
OpenNebula
secure enough to be
OpenStack Nimbus
Amazon
registered
Image Customized for the selected
Infrastructure
• The process of registering
Security Check
an image only needs to be
done once per infrastructure Upload Image to the Infrastructure
Register Image in the
Infrastructure
https://portal.futuregrid.org
Retrieve from
Image Repository
Image is Ready
for Instantiation in
the Infrastructure
Tests Results obtained from the
Analysis of the Image
Management Framework
https://portal.futuregrid.org
Methodology
• Software deployed on the FutureGrid India cluster
– Intel Xeon X5570 servers with 24GB of memory
– Single drive 500GB with 7200RPMm 3Gb/s
– Interconnection network of 1Gb Ethernet
• Software Client is in India’s login node
• Image Generation supported by OpenNebula
• Image Repository supported by Cumulus (store
images) and MongoDB (store metadata)
• HPC supported by xCAT, Moab and Torque
• Performed different tests to evaluate the Image
Generation and the Image Registration tools
https://portal.futuregrid.org
Scalability of Image Generation I
• Concurrent requests to create CentOS images from scratch
• Increasing number of OpenNebula compute nodes to scale
1200
1 Compute Node
1000
2 Compute Nodes
4 Compute Nodes
Time (s)
800
600
400
200
0
1
2
4
Number of Concurrent Requests
http://futuregrid.org
8
Scalability of Image Generation II
• Analyze how the time is spent within the
image creation process
• Only one OpenNebula compute node to better
analyze the behavior of each step of the
process
• Concurrent requests to create CentOS and
Ubuntu images
• Image creation performed from scratch and
reusing a base image from the repository
https://portal.futuregrid.org
Create Image from Scratch
1400
(4) Upload It to the Repository
(3) Compress Image
(2) Generate Image
(1) Boot VM
1200
CentOS
Time (s)
1000
800
600
400
200
0
1
1400
1000
Time (s)
4
8
4
8
(4) Upload It to the Repository
(3) Compress Image
(2) Generate Image
(1) Boot VM
1200
Ubuntu
2
Number of Concurrent Requests
800
600
400
200
0
1
2
Number of Concurrent Requests
https://portal.futuregrid.org
Create Image from Base Image
1400
(4) Upload it to the Repository
(3) Compress Image
(2) Generate Image
(1) Retrieve/Uncompress base image from Repository
1200
CentOS
Time (s)
1000
800
600
400
200
0
1
2
4
Number of Concurrent Requests
8
1400
(4) Upload it to the Repository
(3) Compress Image
(2) Generate Image
(1) Retrieve/Uncompress base image from Repository
1200
Ubuntu
Time (s)
1000
800
600
400
200
0
1
2
4
Number of Concurrent Requests
https://portal.futuregrid.org
8
Scalability of Image Registration
• Register the same CentOS image in different
infrastructures:
– OpenStack (Cactus version configured with KVM
hypervisor)
– Eucalyptus (2.03 version configured with XEN
hypervisor)
– HPC (netboot image using xCAT and Moab)
• Concurrent registrations in Eucalytpus and
Openstack
• Only one request at a time is allowed for HPC
registration (modifies important parts of the HPC
system)
https://portal.futuregrid.org
Eucalyptus
Time (s)
Register Images on Cloud
900
800
700
600
500
400
300
200
100
0
(3) Upload/Register Image into Cloud Infrastructure
(2) Retrieve Image from Server Side
(1) Customize Image
OpenStack
Time (s)
1
900
800
700
600
500
400
300
200
100
0
2
4
Number of Concurrent Requests
8
(3) Upload/Register Image into Cloud Infrastructure
(2) Retrieve Image from Server Side
(1) Customize Image
1
2
http://futuregrid.org
Number of Concurrent
4
Requests
8
Register Image on HPC
140
120
(4) Packimage (xCAT)
Time (s)
100
(3) Retrieve Kernels and Update
xCAT Tables
(2) Uncompress Image
80
60
40
(1) Retrieve Image from
Repository
20
0
1
Number of Concurrent Requests
https://portal.futuregrid.org
Conclusions I
• We have introduced the FG user-controlled
image management framework to handle
images for different infrastructures
• Framework abstracts the details of each
underlying system
• Users can easily create and manage
customized environments within FG
• Replicate software stack on the supported
cloud and bare-metal infrastructures
https://portal.futuregrid.org
Conclusions II
• Image management results show a linear increase
in response to concurrent requests
• Image Generation
– Create image from scratch in only 6 min and using a
base image in less than 2 min
– Scale by adding more nodes to the cloud
– Support different OS and arch due to virtualization
• Image Registration registers images in any
supported infrastructure in less than 3 min
• Image Repository supports perfectly the rest of the
framework with a negligible overhead
https://portal.futuregrid.org
Ongoing Work
• Integrate a messaging queue system (like
RabbitMQ or ZeroMQ) to process user’s
requests in an asynchronous way
• Develop a portal interface
• On-demand resource re-allocation between
infrastructures (usage, user’s requests)
https://portal.futuregrid.org
Thank for your attention!!
Contact info:
Javier Diaz: javier.diazmontes@gmail.com
Gregor Laszewski: laszewski@gmail.com
http://futuregrid.github.com/rain/
https://portal.futuregrid.org
https://portal.futuregrid.org