Jenkins Scalability Summit
Oct 2013
©2013 CloudBees, Inc. All Rights Reserved
1
Logistics
• Where is …?
• Network
©2013 CloudBees, Inc. All Rights Reserved
2
Agenda Part 1: Story time
•
•
•
•
Learn from our collective experience
Identify high-priority problems
What is hurting serious users?
Concrete details are good
©2013 CloudBees, Inc. All Rights Reserved
3
Agenda Part 2: Discuss & Design
• Collectively pick a few topics
• Split into 2 tracks
• See if we can start shaping up solutions
• Please keep shared notes
– Details!
©2013 CloudBees, Inc. All Rights Reserved
4
Possible topics includes …
•
•
•
•
•
•
•
•
•
CPU/memory/disk/network/… consumption
Stability/diagnosability of slaves
Access control of builds/UI
Organizing jobs and build records
Master to master communication
Workflow / choreography
User interface
Stability of Jenkins releases
Plugin compatibilities
©2013 CloudBees, Inc. All Rights Reserved
5
Goals
• No one goes listen-only mode
• Produce notes to show to the broader
community
• Figure out how to do this better next year
©2013 CloudBees, Inc. All Rights Reserved
6
What We’ve Done
&
What We Can
©2013 CloudBees, Inc. All Rights Reserved
7
Thread Reduction
• Per-slave consumption
– SSH (down to 1)
– Channel (down to 1)
• Executor thread on demand
©2013 CloudBees, Inc. All Rights Reserved
8
Thread Reduction: TODO
• NIO
• Asynchronous job execution
– More about this in workflow
©2013 CloudBees, Inc. All Rights Reserved
9
Memory reduction
• Lazy loading of build records
• Database plugin
©2013 CloudBees, Inc. All Rights Reserved
10
Design Choices That Constrain Us
• Compatibility
– Adding is doable, changing is often hard
• Heterogeneousness in data model
• Thread-driven execution model
©2013 CloudBees, Inc. All Rights Reserved
11
CloudBees Scalability Efforts
©2013 CloudBees, Inc. All Rights Reserved
12
Multi-master / Meta Jenkins
• Master-to-master channel
• Extensions on top of it
– “Cloud” impl to lease slaves
– Push security realm
– Push update center
–…
©2013 CloudBees, Inc. All Rights Reserved
13
Multi-master
• Where we think we can go with this
– Move jobs around
– Sharding with UI mashup
• Where this wouldn’t take us
– Loss of inflight builds
– True horizontal scaling
©2013 CloudBees, Inc. All Rights Reserved
14
Workflow
• Brand new job type
• Kill multiple birds in one stone
• Inspired by buildflow & jenkow
©2013 CloudBees, Inc. All Rights Reserved
15
Ingredients
• No single thread to execute the whole thing
– Analogous to NIO
– Check pointing
• BPMN-like workflow execution model
– Surface syntax independent
• Groovy DSL
– With continuation-passing style execution model
• Post-execution visualization
– But no pre-execution visualization
©2013 CloudBees, Inc. All Rights Reserved
16
Ingredients
• Unified properties, build variables, and
environments
• Open up direct access to file store
• Hands-free process forking
• Interop with existing job types
©2013 CloudBees, Inc. All Rights Reserved
17