CloudBroker Platform
Presentation
Wibke Sudholt
CloudBroker GmbH
Technoparkstrasse 1, CH-8005 Zurich, Switzerland
Phone: +41 44 633 79 34
Email: wibke.sudholt@cloudbroker.com
Web: http://www.cloudbroker.com
SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481
CloudBroker Platform
• Web application store for the deployment and
execution of scientific and technical software in the
cloud
• On demand, pay per use, browser / programmatic /
command-line access, cross-domain
• Uses infrastructure as a service (IaaS) from cloud
providers
• Offers platform as a service (PaaS) for software vendors
and software as a service (SaaS) for end users
• Easy to use, speeds up time to market, no need for
own high performance computing infrastructure, OpEx
instead of CapEx
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 2
Platform Architecture
End Users, Software Vendors
User Tools
REST Web Service API
Biology
Applications
Health
Applications
Engineering
Applications
CloudBroker Platform
Amazon
Cloud
© CloudBroker GmbH
All rights reserved.
IBM
Cloud
…
Applications
CloudBroker Integration
Chemistry
Applications
Java Client
Library
CLI
Web
Browser
UI
…
Cloud
SCI-BUS Summer School 2012
05.07.2012 / 3
Platform Functionality
Process
Manager
Users
Tools
Web Browser UI
Web Service API
Application
Manager
Process
Monitor
User
Manager
Queuing
System
Accounting
Module
Resource
Manager
Storage
Manager
Billing
Module
Payment
Module
Image
Manager
Scalability and Fault Tolerance Handler
Cloud Provider Access Manager
Amazon
Adapter
IBM
Adapter
…
Adapter
Security Frame: Transport Layer Security, Access Rights Security
Amazon
Cloud
© CloudBroker GmbH
All rights reserved.
IBM
Cloud
…
Cloud
SCI-BUS Summer School 2012
05.07.2012 / 4
Platform Security
Customer
Client Browser
or Application
Corporate
IT
Corporate Security
Policies and Standards
SSL Secured Connection
Authentication
CloudBroker
CloudBroker
CBP
Platform
.
Industry Standard
Application Security
Technology
Industry
Standard
Server Security
Technology
Industry
Standard
Secure
Data
Center
SSL Secured Connection
SSL secured connection
Authentication to Cloud
Authentication to VM
Cloud Provider
© CloudBroker GmbH
All rights reserved.
Cloud CBP
Instances
Dedicated, Secured
.
and Restricted
Virtual Machines
Security Certified
Compute and
Storage Cloud
Technology
Security
Certified
Data
Center
SCI-BUS Summer School 2012
05.07.2012 / 5
User Management
• User roles: admin, advanced, standard
• Software visibility: private, protected, public
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 6
Typical Job
Submission Lifecycle
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
Prepayment (user) or monthly invoice (platform)
Software and resource selection and job creation (user)
Data file upload (user) to cloud storage (platform)
Job submission (user)
Compute instance startup or reuse (platform)
Data file upload from cloud storage to master node (platform)
Computations on worker nodes (platform, application)
Data file download from master node to cloud storage (platform)
Compute instance shutdown or reuse (platform)
Data file download (user) from cloud storage (platform)
Accounting and billing (platform)
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 7
Dynamic Cluster in the Cloud
• Communication to and between nodes via SSH
• Instance information in dynamically generated node file (MPICH / MPICH2
/ Open MPI format, more formats can be added upon request)
• File sharing via NFS can be enabled
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 8
Application Requirements
Item
Requirements / possibilities
Type
Scientific and technical applications
Domain
All
License
Open source, free, commercial
Character
Compute-intensive, batch-oriented
Interface
Command line, non-interactive
Runtime
Hours, days, weeks
Operating system
Linux
Parallelization
Single-threaded / single-core, multi-threaded / multi-core,
embarrassingly parallel / high-throughput, parallel across
nodes (via MPI or other methods)
Workflows
Single jobs, pipeline jobs, parameter sweeps (via API)
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 9
Ported Applications
Application
Domain
Remarks
BLAST
DNA and protein sequence alignment
Available in public platform
AutoDock
Protein-ligand docking
Available in public platform
Gromacs
Molecular dynamics simulations
Available in public platform
X! Tandem
Mass spectrometry data matching
Available in public platform
OpenFOAM
Computational fluid dynamics
Available in public platform
Rosetta
Protein modelling
Only with own license
TransAT
Computational fluid dynamics
Only with own license
Siesta
Material science
Only with own license
T-Coffee
DNA and protein sequence alignment
FoldX
Protein modelling
R
Mathematics and statistics
More applications continuously to be added
Applications can also be added by users
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 10
Typical Software
Deployment Lifecycle
1.
2.
3.
4.
Generation of installation script (vendor)
Creation and configuration of software (vendor)
Adding of executables (vendor)
Adding of installation package (vendor), including installation
script and software package, if needed
5. Deploying of software (vendor)
6. Checking of software and installation script (manager)
7. Generation of deployments (manager)
8. Running of special deployment jobs (platform)
9. Checking and activation of deployments (manager)
10. Activation of software (manager/vendor)
11. Publication of software (vendor)
12. Running of jobs (user)
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 11
Clients and APIs
CLI
• Command line interface
• Job submission
Java
• Application programming interface
• Java client library
REST
• Representational state transfer
• Web service interface
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 12
Public Platform
• https://platform.cloudbroker.com
• First preview version published in fall 2010, current version 1.0.4
• Available clouds:
– Amazon EC2 and S3 (computing and storage)
– IBM SmartCloud Enterprise (only computing currently, storage upcoming)
• Available applications:
– BLAST, AutoDock, Gromacs, X! Tandem, OpenFOAM
– Additional applications privately or upon request
• Software checking before deployment
• Standard pricing:
– Cloud resource costs + software costs + CloudBroker surcharges
• Production version, monitored and maintained
• Help desk under platform@cloudbroker.com
• Also available as hosted and in-house version
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 13
Protein Modelling Example Case
•
Setup:
– Collaboration between IBM,
CloudBroker and the Institute of
Molecular Systems Biology (IMSB)
at ETH Zurich
– Goal is to better understand the
mechanisms of infectious diseases
– For the example of the Streptococcus bacterium, predict the 3D
structures of important proteins
from their 1D sequences by means
of the Rosetta software
– Use the CloudBroker Platform to
manage the calculations on top of
IBM SmartCloud Enterprise
compute and storage resources
© CloudBroker GmbH
All rights reserved.
•
Results:
– 249 target proteins modeled
– More than 1’000 CPUs used in
parallel
– 36’000 jobs automatically
managed
– Almost 250‘000 CPU hours
consumed
– 2.3 Mio 3D protein structure
models created
– Less than two weeks runtime
SCI-BUS Summer School 2012
05.07.2012 / 14
Further Examples
• ASCOMP TransAT (http://www.ascomp.ch)
– Computational fluid dynamics
– Collaboration with HSR
– Microsoft Silverlight-based web browser UI
• KNIME (http://www.knime.org)
– Konstanz Information Miner
– Workflow framework
– Computational chemistry example case
• SCI-BUS (http://www.sci-bus.eu)
– SCIentific gateway Based User Support
– EU FP7 project
– 11+6 user communities from different domains
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 15
CloudBroker in SCI-BUS
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 16
Goals in SCI-BUS
• Provide connection to open and commercial cloud
infrastructures for the scientific gateways
• Allow scientific gateways to charge for their services
via the CloudBroker Platform
• Integrate CloudBroker Platform as new distributed
computing infrastructure (DCI) in WS-PGRADE / gUSE
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 17
Features to Be Implemented
• Already available:
– Development and testing platform for SCI-BUS
– Extended CloudBroker Platform REST and Java APIs
– Integration of CloudBroker Platform in WS-PGRADE / gUSE
under DCI Bridge, security and job settings
– Possibility to configure and run CloudBroker jobs from within
WS-PGRADE / gUSE workflows
• In preparation:
– Commercial components in WS-PGRADE / gUSE for user
mapping, billing and cost display
– Possibility to register own cloud resources in the CloudBroker
Platform
– Adapters to open cloud infrastructures (OpenStack,
OpenNebula, Eucalyptus)
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 18
Acknowledgements
• CloudBroker GmbH:
– Maxim Malgin, Ann Mikhailova
• ScaleTools AG:
– Nicola Fantini, Andrey Sereda
• MTA SZTAKI:
– Peter Kacsuk, Zoltan Farkas, Akos Hajnal
• … and many others
© CloudBroker GmbH
All rights reserved.
SCI-BUS Summer School 2012
05.07.2012 / 19
Thank you!
Questions?
Please add yourself to the tutorial user list!
SCI-BUS is supported by the FP7 Capacities Programme under contract no. RI-283481