Continuous Integration and
Deployment using
OpenStack
Miguel Zuniga
@mikezuniga
mzigavzquez@ebay.com
May 5, 2014
About ebay inc
eBay Inc. enables commerce by delivering flexible and
scalable solutions that foster merchant growth.
With 145 million active buyers globally, eBay is one of the world's
largest online marketplaces, where practically anyone can buy and
sell practically anything.
With 148 million registered accounts in 193 markets and 26
currencies around the world, PayPal enables global
commerce, processing almost 8 million payments every day.
eBay Enterprise is a leading provider of commerce
technologies, omnichannel operations and marketing solutions.
It serves 1000 retailers and brands.
Agenda
• Enter Fluo (CI/CD)
• Fluo Flow
• Fluo Code Replication
• Packages, Artifacts, <your term here>
• Distribution of Packages
• Infrastructure as Code
• Deployment to Production
• Screenshots
• Roadmap
Continuous Integration and Deployment using OpenStack
3
Enter Fluo (CI/CD)
From begin to end… how code goes to deployment.
At the beginning
Basic system running
• Gerrit
• Jenkins
• Web server
Requirements
• Single Interface
• Scalable
• Simple to use
• Developer Friendly
• Generic
Continuous Integration and Deployment using OpenStack
5
Fluo and the CICD configuration file
• What is Fluo?
– WebUI and API
– Single pane of glass
– Cloud instance prov and decomm
– Allows user’s to configure the system
– RBAC
• How it works?
– Jenkins through API
– Gerrit through API
– Uses ZMQ to communicate with workers which do non-API actions
– Recycle slaves or provisions them on the fly at the moment of job execution.
• What is the cicd configuration file?
– Specification and requirements files
– How the developers instruct the system what to do (run unittests, how to build packages, install
dependencies, more…).
• Example
Continuous Integration and Deployment using OpenStack
6
Fluo cicd configuration file
language: bash
package_install:
apt:
- build-essential
- ruby1.9
- ruby1.9-dev
- mysql-server
- mysql-dev
gem:
- rails -v=4.0.3
- zmq
- mysql2
before_review:
- /etc/init.d/mysql-server start
- mysql -u root < database.sql
- RAILS_ENV=test rake db:migrate
- RAILS_ENV=test rake db:test:load
review_script:
- rake test test/models/user_test.rb
- rake test test/models/session_test.rb
- rake test test/models/post_test.rb
notification:
email:
- myemailuser1@email.com
Continuous Integration and Deployment using OpenStack
7
Fluo
Components
•
•
•
•
•
•
•
•
•
•
Architecture
Fluo (App/worker/coordinator)
Cloud for provisioning
instances
Gerrit
Jenkins
Zuul
Galera (MySQL)
Zabbix (or any other
monitoring)
Puppet (or any other CM tool)
Mrepo
Rsync
Continuous Integration and Deployment using OpenStack
8
Fluo Flow
A step by step journey of a code change.
Basic Workflow
• Fluo basic workflow has 6 different stages/step
– Review
– Approval
– Build
– Integration
– Release
– Periodic
• At each stage/step Fluo will:
1. Build a cloud instance
2. Read the configuration file
3. Execute the scripts defined
4. Report back status
5. Destroy the instance
Developer
Fluo
Commit and request
review
Execute Review Scripts
Approve the code change
Execute Approval Scripts
Merge and replicate code
Execute Build Scripts
Adds “Run Integration” to
comment history on
specific change
Execute Integration
Scripts
Tag’s a specific commit
Execute Release Scripts
At XY time of day
Execute Periodic Scripts
• Customs workflows can be created by users.
Continuous Integration and Deployment using OpenStack
10
Code Quality = Must have… Review, Approval, Build
Review
Approval
Build
• Code change submitted
for review
• Executed before
merging the code when
code change is
approved by someone
• Executed after the code
change has been
merged
The rest of the flow… Integration… Release…
Periodic.
Integration
• Activated by user through
Fluo by commenting “Run
Integration” on a specific
code change.
Release
Periodic
• Activated by user
adding tags to a
specific commit.
• Activated by at a
specific time of the day,
week. Similar to a cron
job.
Continuous Integration and Deployment using OpenStack
11
Fluo Code Replication
One project… 2 projects… Multiple projects.. Best
Practices.
Github, Internal Github and your own Git repository.
Some points to consider:
• Categorize your code
– Will it be open sourced?
– Is it private for our team?
– Is it private for our company?
– Do we need mirrors?
Github
• Always use ssh keys
• Standards
– User which will be replicating
– Replicate Branches or maybe not…
Fluo
Private
Github
Local
Github
Continuous Integration and Deployment using OpenStack
13
Packages, Artifacts, <your term here>
How do you want to wrap it out? RPM? Debs? Tar?
When and where do I create my packages?
Things to consider.
• Jenkins slaves will build the packages.
• Make sure all your dependencies are in your cicd config file.
• Prepare your environment with the package_install and before_{stage} sections.
• Build your package at the {stage}_script section.
• Define a versioning standard (0.0.x for test packages, x.y.0 for production packages)
Two main stages were created for build purposes.
• Build stage
– Use commit number + time in secs as package version.
• Release stage
– Use git tag as package version.
Continuous Integration and Deployment using OpenStack
15
Distribution of Packages
Shipping the code to different locations.
Shipping Packages to a Central Point
• Use simple and secure methods.
– Secure Object storage
– Rsync + SSH
– SCP
• Mrepo (if you are RPM Based)
• Replication is done every 5 mins.
Replicate… Replicate and Replicate.
Cloud A
Datacenter
1
Fluo
Package
Repository
Datacenter
2
Cloud B
Continuous Integration and Deployment using OpenStack
17
Infrastructure as Code
Controlling and Managing what is your Infrastructure
doing.
Configuration Management
• At the moment we use Puppet.
• Puppet Code will also go through Fluo.
Some guidelines to our puppet coders.
•
•
•
•
•
•
•
Create your CM code in a service oriented way.
Use virtual resources.
Create within a module sections for each type
of OS you work with.
Create a common module.
Use parameterized classes.
Puppet parser validate filename.pp at the
review/approval stage.
Puppet apply –vd --noop -modulepath=/modules filename.pp at the
review/approval stage.
Example:
• virtual.pp
• classes
• someclass.pp
• modules
• Fluo
• files
• templates
• manifests
• init.pp
• Redhat
• install.pp
• config.pp
• service.pp
• Ubuntu
• install.pp
• config.pp
• postconfig.pp
• service.pp
Continuous Integration and Deployment using OpenStack
19
Deployment to Production
So it’s GO time and you have green light.
It’s GO time
• Centralize your configuration management code
• Keep it in sync (pull from the repository every 2 or 5 mins).
• Tag your Releases with a chronological meaning and keep a standard
– OK
• 1.1.0
• 1.2.0
• 1.10.0
– NOT OK
• V1.0
• Beta1.0.1
• 1.2
• Remember the TAG goes into the name of the package and its how your package manager identifies
which package should be installed on an upgrade.
• Automatic Deployment or Scheduled Deployment?
Continuous Integration and Deployment using OpenStack
21
Some Fluo screens
Continuous Integration and Deployment using OpenStack
23
Continuous Integration and Deployment using OpenStack
24
Continuous Integration and Deployment using OpenStack
25
Continuous Integration and Deployment using OpenStack
26
Roadmap
The Future of Fluo at eBay Inc.
• CICDaaS (CICD as a Service).
• Integrate with other CM tools (chef, salt).
• Completely control of CM tools through Fluo.
• Container support (OpenVZ/Docker).
• Add block storage as option to store packages/artifacts.
• Ironic, Cobbler, Foreman, Razor integration.
• Jenkins? Zuul?
• Improve UI.
Continuous Integration and Deployment using OpenStack
28
Thank you.
Yes. We are hiring!
dl-ebay-cloud-hiring@corp.ebay.com