Creating a Multi-Cloud System
with the CloudBroker Platform
Dr. Wibke Sudholt
CloudBroker GmbH
Räffelstrasse 25, CH-8045 Zürich, Switzerland
Phone: +41 44 515 21 70
Email: info@cloudbroker.com
Web: http://www.cloudbroker.com
SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481
Overview
• Cloud computing
• CloudBroker Platform
• SCI-BUS project
• Further examples
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 2
Cloud Computing
SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481
Cloud Definition
• According to the US
National Institute of
Standards and Technology
(NIST)
• Essential characteristics
–
–
–
–
–
On-demand self service
Broad network access
Resource pooling
Rapid elasticity
Measured service
© CloudBroker GmbH
All rights reserved.
• Service models
– Software as a Service
(SaaS)
– Platform as a Service
(PaaS)
– Infrastructure as a Service
(IaaS)
• Deployment models
–
–
–
–
Private cloud
Community cloud
Public cloud
Hybrid cloud
SCI-BUS Summer School 2013
02.07.2013 / 4
What is new?
• Combination of previously existing
technologies
• Change in IT business model, OpEx instead of
CapEx
• Outsourcing, automation, service orientation
and economy of scale
• Interfaces between providers and users set at
the right place
• Wholesale instead of handcrafted version of IT
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 5
Advantages of Cloud
•
•
•
•
•
•
•
•
Immediate, on demand access
Availability of otherwise unreachable resources
Nearly unlimited scalability
Dynamic load balancing, cloud bursting
No long-term binding, flexibility
User self service
Pay-per-use with minimal initial investment
Focus on core competencies
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 6
Challenges of Cloud
• Hype, cloud washing
• Complexity and dynamics
• Performance limitations, dependency on
network
• Standardization and vendor lock-in issues
• Security, privacy and legal concerns
• New service and business model
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 7
Clouds for Scientific Computing
• Lots of computing resources and specialized application
software needed
• Resources often either not existing at all or at capacity
limit
• Hardware and software expensive, time-consuming and
complex to buy, set up, use and maintain
• Mostly project-based or periodical usage patterns
⇒ Ideal case for cloud computing
⇒ Clouds make large-scale scientific computing resources
also accessible for SMEs, small research groups, niche
applications and short-term projects
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 8
Science Gateways and Clouds
• According to the EGI-InSPIRE Science Gateway
Primer
• Three principle approaches by which science
gateways can interact with cloud computing
– Utilize cloud computing resources for the services
they offer
– Run in the cloud themselves
– Adopt the cloud business model
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 9
Scientific Computing Cloud Stack
User
Gateway
SaaS
PaaS
IaaS
HPC
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 10
CloudBroker Platform
SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481
CloudBroker Platform
• Web application store for the deployment and
execution of compute-intensive scientific and
technical software on different clouds
• Uses IaaS from cloud providers
• Offers PaaS for software vendors and SaaS for
end users
• On demand, pay-per-use, browser, programmatic
and command-line access, cross-domain
• Easy to use, no need for own scientific computing
infrastructure, speeds up time to results, OpEx
instead of CapEx
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 12
Platform Cloud Stack
User
Gateway
SaaS
PaaS
CloudBroker
Platform
IaaS
HPC
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 13
Platform Cloud Stack
User
Gateway
SaaS
PaaS
CloudBroker
Platform
IaaS
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 14
Platform Architecture
End Users, Software Vendors, Resource Providers
CLI
Java Client Library
REST Web Service API
Chemistry
Applications
Biology
Applications
Pharma
Applications
Engineering
Applications
…
Applications
CloudBroker Platform
Amazon
Cloud
© CloudBroker GmbH
All rights reserved.
IBM
Cloud
OpenStack
Cloud
OpenNebula
Cloud
Eucalyptus
Cloud
CloudBroker Integration
Web
Browser
UI
User Tools
…
Cloud
SCI-BUS Summer School 2013
02.07.2013 / 15
Platform Functionality
Process
Manager
Users
Tools
Web Browser UI
Web Service API
Application
Manager
Process
Monitor
User
Manager
Queuing
System
Accounting
Module
Resource
Manager
Billing
Module
Storage
Manager
Payment
Module
Image
Manager
Scalability and Fault Tolerance Handler
Cloud Provider Access Manager
Amazon
Adapter
IBM
Adapter
OpenStack
Adapter
OpenNebul
a
Adapter
Eucalyptus
Adapter
…
Adapter
Security Frame: Transport Layer Security, Access Rights Security
Amazon
Cloud
© CloudBroker GmbH
All rights reserved.
IBM
Cloud
OpenStack
Cloud
OpenNebula
Cloud
Eucalyptus
Cloud
…
Cloud
SCI-BUS Summer School 2013
02.07.2013 / 16
Available Cloud Adapters
• Virtual machines
–
–
–
–
–
Amazon EC2
IBM SmartCloud Enterprise
OpenStack EC2 and Nova
OpenNebula
Eucalyptus
• Object storage
–
–
–
–
Amazon S3
IBM Nirvanix
OpenStack S3
Ceph RADOS
⇒ More cloud adapters continuously to be added
⇒ Users can use and pay for public resources registered by
CloudBroker and others
⇒ Users can register and charge for their own resources, just
cloud compute and storage accounts needed
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 17
Platform Security
Customer
Client Browser
or Application
Corporate
IT
Corporate Security
Policies and Standards
SSL Secured Connection
Authentication
CloudBroker
CloudBroker
CBP
Platform
.
Industry Standard
Application Security
Technology
Industry
Standard
Server
Security
Technology
Industry
Standard
Secure
Data
Center
SSL Secured Connection
SSL secured connection
Authentication to Cloud
Authentication to VM
Cloud Provider
© CloudBroker GmbH
All rights reserved.
Cloud CBP
Instances
Dedicated, Secured
.
and Restricted
Virtual Machines
Security Certified
Compute and
Storage Cloud
Technology
Security
Certified
Data
Center
SCI-BUS Summer School 2013
02.07.2013 / 18
Interaction of Tiers
Object
Storage
Worker
Node 1
Client
Platform
Master
Node
Worker
Node 2
Image
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 19
User, Resource and Software
Management
CloudBroker
Platform
Organization 1:
Resources,
Software, Invoices
User 1.2:
Jobs, Data Files,
Instances
User 1.2:
Jobs, Data Files,
Instances
Organization 2:
Resources,
Software, Invoices
User 2.1:
Jobs, Data Files,
Instances
User 2.2:
Jobs, Data Files,
Instances
• User roles: admin, advanced, standard
• Resource and software visibility: private, protected, public
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 20
Application Requirements
Item
Requirements / Possibilities
Type
Scientific and technical applications
Domain
All
License
Open source, free, commercial
Character
Compute-intensive, batch-oriented
Interface
Command line, non-interactive
Runtime
Hours, days, weeks
Operating system
Linux
Parallelization
Single-threaded / single-core, multi-threaded / multi-core,
embarrassingly parallel / high throughput, parallel across
nodes (via MPI or other methods)
Workflows
Single jobs, pipeline jobs, parameter sweeps (via API)
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 21
Ported Applications
• Chemistry
– Gromacs, Siesta
• Biology
– BLAST, X! Tandem, Rosetta, TCoffee, FoldX, ClustalW, Clustal
Omega
• Pharma
– AutoDock
• Engineering
– OpenFOAM, ASCOMP TransAT,
Elmer
• Rendering
– Blender
• General
– Shell scripts, R, Akka, PHP,
wrapper script
⇒ More application software continuously to be added
⇒ Users can use and pay for public software deployed by CloudBroker and
others
⇒ Users can deploy and charge for their own software, just installation script
and software package needed
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 22
Clients and APIs
CLI
• Command line interface
• Job submission
Java
• Application programming interface
• Java client library
REST
• Representational state transfer
• Web service interface
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 23
Public Platform
• Public CloudBroker Platform version under
https://platform.cloudbroker.com
• Free organization and user account after registration and
validation
• Standard pricing = (resource charges + software charges) ×
(1 + CloudBroker surcharge percentage)
• Freemium model: CloudBroker surcharges only for
commercial and pay-per-use cloud resources and
application software
• Production version: Monitored, maintained and regularly
updated
• Helpdesk under platform@cloudbroker.com
• Also available as hosted and licensed versions
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 24
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 25
SCI-BUS Project
SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481
SCI-BUS Project Overview
• Project
– SCI-BUS = SCIentific gateway
Based User Support
– Supported by the EU FP7
Capacities Programme under
contract no. RI-283481
– Running 10/2011-09/2014
– http://www.sci-bus.eu
• Partners
– 15 consortium members from
10 different countries (10
academic, 5 commercial)
– 6 subcontractors
– Growing number of associated
partners (signing of MoU)
© CloudBroker GmbH
All rights reserved.
• Goals
– Build science gateways for
different communities and
domains
– Provide researchers seamless
access to major computing,
data and networking
infrastructures and services
– Focus on scientific workflows
– Develop corresponding
gateway technology
SCI-BUS Summer School 2013
02.07.2013 / 27
CloudBroker’s Role in SCI-BUS
• Provide connection to public and private,
commercial and academic, proprietary and open
cloud infrastructures via the CloudBroker
Platform
• Support the integration of the CloudBroker
Platform as new distributed computing
infrastructure (DCI) in WS-PGRADE/gUSE
• Enable science gateways to charge for their
services via the CloudBroker Platform
• Lead the project’s efforts towards sustainability
and development of business models
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 28
SCI-BUS Cloud Stack
User
Gateway
SCI-BUS
SaaS
PaaS
CloudBroker
Platform
IaaS
HPC
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 29
SCI-BUS Cloud Stack
User
Gateway
WS-PGRADE/
gUSE
SaaS
PaaS
CloudBroker
Platform
IaaS
HPC
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 30
SCI-BUS Architecture
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 31
Current Clouds in SCI-BUS
• Public commercial clouds
– Amazon Web Services
– IBM SmartCloud Enterprise
• Private academic clouds
–
–
–
–
MTA SZTAKI, Hungary: OpenNebula and Ceph RADOS
BIFI, University of Zaragoza, Spain: OpenStack
ETH and University of Zurich, Switzerland: OpenStack
University of Westminster, UK: OpenStack
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 32
WS-PGRADE/gUSE
Source: http://www.guse.hu/about/architecture
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 33
CloudBroker Platform
Integration in WS-PGRADE/gUSE
•
Methodology
–
–
–
•
Implemented features
–
–
–
–
–
•
CloudBroker DCI Bridge configuration
portlet
CloudBroker security portlet
Concrete workflow portlet extension for
CloudBroker jobs
Support for running own applications via a
wrapper script
Display of costs
Upcoming features
–
•
Collaboration between MTA SZTAKI,
CloudBroker and ScaleTools
Use of the CloudBroker Platform Java
client library
Coordination of updates
Display of prices and invoices
Future features
–
–
Different types of user mapping
Data Bridge integration
Source: http://www.guse.hu/node/49
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 34
Current SCI-BUS Gateways
•
•
•
•
•
•
•
•
•
•
•
Adria Science Portal
AEGIS CMPC Portal
agINFRA Science Gateway
AutoDock Portal
BIFI-Unizar Portal
Citizen Web Community
Gateway
AMC e-BioInfra Gateway
Etics Portal
iportal
MoSGrid Portal
NVG Science Gateway Portal
© CloudBroker GmbH
All rights reserved.
•
•
•
•
•
•
•
•
•
•
Public gUSE Portal
RenderFarm.fi
SHIWA Simulation Platform
SimBusPro Portal
Statistical Seismology
Science Gateway
TR-Grid gUSE Gateway
VisIVO Gateway
HELIOGate Portal
Westminster DG Portal
WRF4SG
SCI-BUS Summer School 2013
02.07.2013 / 35
Further Examples
SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481
Protein Modelling
•
Setup
– Collaboration between IBM,
CloudBroker and the Institute of
Molecular Systems Biology (IMSB)
at ETH Zurich
– Goal is to better understand the
mechanisms of infectious diseases
– For the example of the Streptococcus bacterium, predict the 3D
structures of important proteins
from their 1D sequences by means
of the Rosetta software
– Use the CloudBroker Platform to
manage the calculations on top of
IBM SmartCloud Enterprise
compute and storage resources
© CloudBroker GmbH
All rights reserved.
Source: Dr. Lars Malmström,
IMSB, ETH Zurich
•
Results
– 249 target proteins modeled
– More than 1’000 CPUs used in
parallel
– About 36’000 jobs automatically
managed
– Almost 250‘000 CPU hours
consumed
– 2.3 Mio 3D protein structure
models created
– Less than two weeks runtime
SCI-BUS Summer School 2013
02.07.2013 / 37
Interfacing in KNIME
• KNIME = Konstanz
Information Miner
• Open-source data
integration, processing,
analysis and exploration
framework supported by
KNIME.com AG
• Experimental CloudBroker
Platform KNIME node
• Computational chemistry
workflow showcase:
Calculate the dipole
moments for a number of
molecules
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 38
Web UI for TransAT
Source: Anita Hollenstein and Patrice Müller,
http://eprints.hsr.ch/193/, p. 59
•
•
•
•
•
TransAT = Transport phenomena Analysis Tool
Computational multiphase fluid dynamics (CMFD) software from ASCOMP GmbH
Collaboration with Hochschule für Technik Rapperswil (HSR), Switzerland
Bachelor thesis “Web UI-Front-End for Fluid Dynamics Cloud”
Microsoft Silverlight-based TransAT web UI
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2013
02.07.2013 / 39
UberCloud HPC Experiment
Heat Transfer Use Case
• Collaboration with Biscarri
Consultoria SL, Spain, CSC IT
Center for Science Ltd., Finland,
and CAELinux, Switzerland
• Team 30 of UberCloud HPC
Experiment
• Application of Elmer finite
element software for
multiphysical problems on a heat
transfer use case
• Using Amazon Cluster Compute
instances via the CloudBroker
Platform
• Results show that scalability is
determined by network backbone
performance
© CloudBroker GmbH
All rights reserved.
Source: Biscarri Consultoria SL, see also
http://tci.taborcommunications.com/UberCloud_HPC_Experiment
SCI-BUS Summer School 2013
02.07.2013 / 40
CloudSME Project
• EU FP7 project
• Led by University of
Westminster, involving 12
SMEs
• Part of the “ICT for
Manufacturing SMEs” (I4MS)
initiative in the Public Private
Partnership “Factories of the
Future” (PPP FoF)
• Starting from July 2013
onwards
• Cloud-based one-stop-shop for
simulations in manufacturing
and engineering
• SaaS and PaaS solutions
© CloudBroker GmbH
All rights reserved.
Source: http://cordis.europa.eu/fp7/ict/computing/home-i4ms_en.html
SCI-BUS Summer School 2013
02.07.2013 / 41
Thank you! – Questions?
For more information
Email: info@cloudbroker.com
Web: http://www.cloudbroker.com
Public platform: https://platform.cloudbroker.com
Helpdesk: platform@cloudbroker.com
SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481