One Click Cloud Orchestrator (OCCO):
bringing Complex Complete
Infrastructures Effortlessly to IaaS
Clouds
Peter KACSUK, Adam VISEGRADI, Jozsef KOVACS and
Márk GERGELY
MTA SZTAKI
Developing an Open Science Backbone
Federated operations and
support
• Service desk
• Monitoring and
accounting
• Capacity management
• Service level
management
• Network of CSIRT
• Federated IdPs,
Auth and Authz
• Management of
different levels of
assurance
• Research platform
built on top of
shared capabilities
plus community
owned resources
• Data products,
tools, scientific
gateways, virtual
labs
Research Infrastructures and
long tail of science
Shared capabilities based on open standards
Multi-level
governance with
community
participation
• Local
• National
• European
Common national pools of resources
Developing an Open Science Commons
From Member States
• Capacity dedicated to
large RIs
• Free pools for long tail
researchers
• Both pubicly funded and
commercial providers (all
supporting open
standards and no lock-in)
Credit to Sergio Andreozzi
Core capabilities
• Open Science
Cloud (e.g., VM
management, Data
storage/access/disc
overy)
• PID
• Service registry
and marketplace
Motivations for OCCO (Example 1)
• Build an institutional
desktop grid for
universities
• Required BOINC
expertise
• Took several weeks
to establish
• Goal: setup the
desktop grid in ~10
minutes in the cloud
• Easy creation of
institutional desktop grids
• Used in IDGF-SP project
Univ. cloud (or EGI cloud)
BOINC
project
server
The whole infrastructure
should be created by 1
click
(Number of BOINC clients
is a parameter)
BOINC
client
BOINC
client
Motivations for OCCO (Example 2)
Build the BOINC infrastructure with a gateway to submit parameter
sweep applications into BOINC
Univ. cloud or EGI cloud
SZTAKI Desktop Grid server
Gateway
(WS-PGRADE)
To run PS applications
on BOINC
3G Bridge
Metajob plugin
BOINC
project
DC-API plugin
GBAC
BOINC
client
Virtualisation
by
GBAC
BOINC
client
Virtualisation
by
GBAC
Motivations for OCCO (Example 3)
Setting up an Autodock SaaS (gateway with BOINC infrastructure):
Univ. cloud or EGI cloud
SZTAKI Desktop Grid server
Autodock
Gateway
3G Bridge
Metajob plugin
BOINC
project
DC-API plugin
GBAC
Biologist, chemist can use it
without any learning curve
BOINC
client
Virtualisation
by
GBAC
BOINC
client
Virtualisation
by
GBAC
Using Autodock SaaS service
E-mail notification of the user
Admin page showing the setup of the service
E-mail notification of the user
The gUSE portal can be reached at
http://dgdemo5.lpds.sztaki.hu/
Motivations for OCCO
• The creation of this 1-click infrastructure took several weeks -> we want a
solution that enables the construction in several days
• There are many other use cases where the goal is to set up a complete
infrastructure on demand in the cloud
• Such an infrastructure typically consists of multiple virtual machine
applications
• These should be instantiated and managed in an automated way
• We need an offline description how such an infrastructure should look like
• Then the infrastructure can be instantiated automatically, or with a single
click of a user. Hence the name: One-Click Cloud Orchestrator
• The infrastructure typically defined as a graph where nodes are
infrastructure services.
• Automation possibilities in a cloud
o Node instantiation
• Through an API or a UI
o Node configuration management
• Via Chef, Puppet, etc.
o Our goal is to extend these to infrastructure
Problems to be solved
• Complex, multi virtual machine applications need special care
that are not offered by current infrastructure clouds
o Users frequently must fine tune their virtual infrastructures manually to meet
their applications needs (scaling, inter-vm dependencies, error resilience)
o Users of multiple clouds are usually restricted to the use of a particular provider
• Users of complex, multi virtual machine applications currently
need low level understanding of application subcomponents
and clouds for:
o Configuration and deployment of subcomponents in a cloud context (this is
partially offered by configuration management tools like Chef or Puppet)
o Such VMs must be crafted for the subcomponents that are capable to interface
with other dynamically created VMs
• We are developing a tool for the EGI communities to provide
easy-to-use management for multi VM applications
State of the Art
feature\System
OS Support
Juju
Ubuntu
OneFlow
Hypervisor dependant
Cloud- Formation
Fixed list of Linux
distributions
Windows Server
FreeBSD
AWS EC2, HP Cloud
Services, Windows
Azure, Openstack
OpenNebula
Node management
method
Service Composer
Image Based
Service Composer
Open source
Yes
Yes
Hosted service
Hosted
Infrastructure Auto
healing
Heat
Hypervisor dependant
SlipStream
Supports wide variety
of OS
"OneClick" plans
"OneClick" plans
31-Jul-14
31-Jul-15
OpsWorks
Amazon Linux,
"OneClick" “vision”
Supports wide variety Supports wide variety Supports wide variety
of OS
of OS
of OS
Ubuntu 12.04 LTS
General EC2,
General EC2,
could be easily
extended later
OCCI,
OpenStack, AWS EC2,
OCCI, Microsoft Azure,
etc.
AWS EC2
Service Composer
Image Based
Service Composer
No
Yes
Partly
No
Yes (?)
Yes (?)
Yes (?)
No
Hosted
No
Hosted
Hosted
No
No
No + SaaS
No
Yes
Yes
Yes
No
Yes
Hopefully
Yes
Yes
Manual Infrastructure
Scalability
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Automatic
Infrastructure Scaling
No
Yes
Yes
Yes
No
Yes
No
Yes
Yes
UI
CLI, GUI
Web frontend, CLI
management console,
CLI, API
CLI, API, Horizon
dashboard
Web UI
management console,
CLI, SDK
API, CLI
API, CLI, HTML5 Web
frontend
API, CLI, HTML5 Web
frontend, SaaS
One Click UI
No
No
No
No
No
No
Yes
Yes
Yes
Service composer
support
Juju
—
Chef, Puppet
Chef, Puppet
—
Chef
loose integration with
Chef
Multi-cloud
provisioning
Yes
No
No
No
Yes
No
Yes
Yes
Yes
Maturity (subjectively
between 1-5)
3
1
5
2
2
1
1
2
4
Supported cloud
backends
AWS EC2
OpenStack
All widely accepted
Cloud Interfaces
could be easily
extended later
Service Composer,
Service Composer,
Chef, other SotA
composer(s)
Chef
Generic Service
Composer, Integrated
Abstract image
management
Generic Service
loose integration with Composer, Integrated
Chef, other SotA
Abstract image
composer(s)
management
OCCO in general
• Instead of basic components like virtual machines, OCCO can
provide complete infrastructures of services in an on-demand,
self-service fashion
• Automatic scaling and error recovery will be included.
• OCCO aims both providing
o a high-level service with UI
o exposing the underlying architecture as a framework to be built upon
• OCCO orchestrates
o resource provider backends (e.g. IaaS clouds)
o and configuration manager backends (e.g. Chef)
based on statically defined infrastructure descriptions (“infrastructure-ascode” paradigm).
• The OCCO framework is developed to be highly versatile and
backend-agnostic.
One Click Cloud Orchestrator (OCCO)
Virtual
Infrastructure
User
use
br
ow
se
Deployment Descriptor
Template Store
OCCO
select
Customized
Infrastructure
Deployment
Descriptor
Virtual
Infrastructure
Node resolution
deploy
request One Click infrastructure
customizer UI
monitor
customize
initiate
monitoring
r
av epor
aila t
bili
ty
Infrastructure State
Notification Sevice
Automated Infrastructure
Maintenance
OCCO
create &
operate
• Analogous to IaaS clouds: users can simply request complex
infrastructures via a simple UI/API.
o Instead of choosing from virtual machine types, users choose from virtual
infrastructure types
o They still face the same pay as you go model, and they can still utilize the
flexibility of the IaaS’s just on larger scale.
View of the OCCO Administrator
Virtual
Infrastructure
User
use
br
ow
se
Deployment Descriptor
Template Store
select
Customized
Infrastructure
Deployment
Descriptor
Virtual
Infrastructure
deploy
request One Click infrastructure
customizer UI
monitor
customize
initiate
monitoring
r
av epor
aila t
bili
ty
Infrastructure State
Notification Sevice
Automated Infrastructure
Maintenance
create &
operate
• OCCO Administrators are expected to create node definitions and
infrastructure descriptions (templates)
o Based on existing configuration management descriptions (e.g. Chef recipes)
o The templates are stored inside the OCCO provided template store in order to
allow reuse
16
View of the OCCO User
Virtual
Infrastructure
User
use
br
ow
se
Deployment Descriptor
Template Store
select
Customized
Infrastructure
Deployment
Descriptor
Virtual
Infrastructure
deploy
request One Click infrastructure
customizer UI
monitor
customize
initiate
monitoring
r
av epor
aila t
bili
ty
Infrastructure State
Notification Sevice
Automated Infrastructure
Maintenance
create &
operate
• Can browse and customize offered infrastructure description templates
• Can receive notifications about the state of his/her infrastructure (to be
implemented)
o Through email or an automated service (allowing immediate use after creation or
changes)
o If a change is applied to the template then the new updated deployment descriptor can
be pushed to OCCO internals
17
Internal Architecture
• Automated Infrastructure
maintenance
o infrastructure descriptor processing and
VM management initiator
Automated Infrastructure
Maintenance
1
A
Infrastructure
Deployment
Descriptor
Compiler
B
Information
Dispatcher
Enactor
2
C
3
4
D
5
• Infrastructure Processor
o internal depiction of a virtual
infrastructure
Infrastructure Processor
!
• Cloud Handler
o abstracts IaaS functionality (e.g., VM
creation) for federated and
interoperable use of clouds
• Service Composer (vm
reshaper)
!
VM Reshaper
A
B
Cloud
Handler
C
D
o ensures awaited functionalities for VMs
• Information Dispatcher
o de-couples the information producer
and consumer roles across the
architecture
D
Cloud 1
B
A
C
Cloud 2
D
C
C
Cloud 3
18
Definition of an infrastructure
Definition of an infrastructure
Deployment example: Wordpress-mysql
Word
press
MySQL
VM1
Cloud-init
Cloud-init
VM2
Deploy
MySQL
Deploy
Wordpress
EC2
Get
MySQL
ipaddress
Infrastructure
description
Instantiate
new VM
Instantiate
new VM
with
MySQL
ipaddress
Register
MySql
CHEF
OCCO
Register
Wordpress
Development environment of OCCO
•
•
•
•
•
In alpha stage
Python 2.7
Git [https://gitlab.lpds.sztaki.hu/groups/cloud-orchestrator]
Jira [https://jira.lpds.sztaki.hu/browse/OCD] – agile sprint dev.
Documentation [http://c153-33.localcloud/util-doc/util.html]
o Sphinx, docstrings: automatic documentation generation
• Testing: Python nosetests: automatic testing
• Deployment: setuptools
o Development package repo: [http://c153-86.localcloud:8080/packages]
o Works best with virtualenv
• Package dependencies
o PyYAML, argparse, python-dateutil, pika (for RabbitMQ), Boto, etc.
• Will be open-source after first release in 2015 Q3
Documentation of OCCO
Conclusions
• Prototype version of OCCO is working
• First version of OCCO is close to be released
• Short term goal: to create OCCI plugin for EGI
FedCloud
• New use cases are continuously developed
• We are looking for partners to realize their use cases