Jenkins Slave Cloud with Apache Mesos
Klaus Azesberger
Reinhard Kiesswetter
Infonova GmbH
Agenda
#jenkinsconf
●
Our Jenkins
●
Reasons to adopt our approach
●
Pains of a static slave cloud
●
Live Demo of setup and common ops use cases
Footer
Standing on the shoulders of giants
#jenkinsconf
●
Vinod Kone
●
eBay PaaS team
Footer
Our Jenkins
#jenkinsconf
●
300 Developers
●
35 + Projects
●
~ 2500 Jobs
●
One Jenkins to rule them all
Footer
Features you need with centralized approach
#jenkinsconf
✔
Jenkins Master High Availability
✔
Maintenance without downtime
✔
100 % reliable pre-prod environment
✔
Rollbacks
✔
Isolation of builds (through reused VMs)
Footer
Should you adopt our approach?
#jenkinsconf
●
Costs of this setup
●
Size (of your Jenkins) doesn't matter
●
SysAdmin background helps
Footer
What we want
#jenkinsconf
●
●
●
A single, unified scalable HW resource pool
(Mesos)...
...to run any kind and size of slaves
Slaves should be instantly available and lightweight
(Docker)
Footer
Setup listing
#jenkinsconf
●
Jenkins Mesos Plugin
●
Mesos Master Cluster (with Zookeeper)
●
Mesos Slaves
●
Jenkins Slave Docker Images
●
CoreOS
●
Monitoring
Footer
Problems with static approach
#jenkinsconf
●
Managing Slave
Instances
●
Installs
●
Updates / Versioning
Backward compatibility
●
Balancing between Labels
●
●
Sizing
●
Footer
Static slaves have a
history
Problems with static approach
#jenkinsconf
●
●
●
Managing Slave Instances
Balancing between
Labels
Slave Slave
Sizing
Slave Slave
Label 2
Slave Slave
Label 3
Slave Slave
Label 4
Slave Slave
Footer
Label 1
Problems with static approach
#jenkinsconf
●
●
●
Managing Slave Instances
Build queue
Balancing between
Labels
Slave Slave
Sizing
Slave Slave
Label 2
Slave Slave
Label 3
Slave Slave
Label 4
Slave Slave
Footer
Label 1
Problems with static approach
#jenkinsconf
●
●
●
Managing Slave Instances
Build queue
Balancing between
Labels
Slave Slave
Sizing
Slave Slave
Label 2
Slave Slave
Label 3
Slave Slave
Label 4
Slave Slave
Footer
Label 1
Problems with static approach
#jenkinsconf
●
Managing Slave Instances
●
Balancing between Labels
●
Sizing
●
Waste of Resources
VS.
●
Footer
Long or even failed builds
Setup listing
#jenkinsconf
●
Jenkins Mesos Plugin
●
Mesos Master Cluster (with Zookeeper)
●
Mesos Slaves
●
Jenkins Slave Docker Images
●
CoreOS
●
Monitoring
Footer
Some Mesos context
#jenkinsconf
●
●
Scheduler
–
Launches tasks
–
Receives resource offers
Executor
–
Executes tasks
–
Launched by the Mesos slave
Footer
Framework
Mesos
#jenkinsconf
Jenkins master
Standby master
Mesos master
Standby master
Mesos slave
Mesos slave
Mesos slave
Docker executor
Docker executor
Docker executor
Jenkins slave
Jenkins slave
Jenkins slave
Footer
Mesos Jenkins Plugin explained
Jenkins Mesos Plugin
Scheduler
cloud
#jenkinsconf
Jenkins Master
Job Queue
Jenkins Slave
(Task)
Mesos master
Job
Mesos slave
Docker executor
18
Footer
#jenkinsconf
M
S
Private
Docker Registry
19
Footer
Sharing is caring
#jenkinsconf
http://github.com/kazesberger/jenkins-mesos-setup
https://github.com/kazesberger/jenkins-docker-slaves
20
Footer
Ansible in short
21
#jenkinsconf
●
Automate setup
●
Automate maintenance
●
Execute commands on a set of hosts
●
yml + python + ssh
●
No agents
Footer
Demo: Use Cases
#jenkinsconf
●
Scale out HW
●
Update SW (firefox)
–
●
One can also provide legacy support
Change Sizing of slaves
–
One can also provide a set of different sizingvariants
Footer
Migration without new server
#jenkinsconf
Build queue
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Footer
Label 1
Label 2
Label 3
Label 4
Jenkins slaves → Mesos slaves
#jenkinsconf
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Footer
Label 1
Label 2
Label 3
Label 4
Load during business hours
#jenkinsconf
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Footer
Label 1
Label 2
Label 3
Label 4
Load during the night
#jenkinsconf
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Slave
Footer
Label 1
Label 2
Label 3
Label 4
Lessons learned
#jenkinsconf
●
●
●
When configuring systemd services keep in mind that
the coreOS node could be autoupdated at any time
When building docker containers for your jenkins slaves
respect settings like Timezone and character encoding
You should do a soft migration to the mesos cloud with
projects that can provide some meaningful feedback in
case of problems
Footer
Klaus Azesberger
#jenkinsconf
Technology Architect
INFONOVA GmbH
A BearingPoint Company
Seering 6
A-8141 Unterpremstätten/Graz
Austria
Tel: +43 316 8003 2106
Fax: +43 316 8003 1080
Mobile: +43 664 81 61 963
www.infonova.com
klaus.azesberger@infonova.com
Reinhard Kiesswetter
Senior System Engineer
INFONOVA GmbH
A BearingPoint Company
32
Seering 6
A-8141 Unterpremstätten/Graz
Austria
Tel:
+43 316 8003 1173
Fax: +43 316 8003 1080
Mobile: +43 664 81 61 886
www.infonova.com
reinhard.kiesswetter@infonova.com
Thank You Sponsors
Footer