Smoothing the continuous delivery path
A tale of two teams
Making Software. Better.
@EqualExperts
@lyndsp
simple software solutions to big business problems.
© Equal Experts UK Ltd and lyndsayp ltd 2015
What’s this all about?
Continuous
Delivery
Done Well
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
2
What’s this (Continuous Delivery) all about?
“... producing valuable software
in
short cycles and ensure that the
software can be
at
@EqualExperts
@lyndsp
reliably released
any time”
https://en.wikipedia.org/wiki/Continuous_delivery
© Equal Experts UK Ltd and lyndsayp ltd 2015
3
Continuous doughnuts
Risk Reduction
T: 0:00:000
@EqualExperts
Fast feedback
@lyndsp
T: 2:00:000
© Equal Experts UK Ltd and lyndsayp ltd 2015
4
There’s no (CD) silver bullet
1 (php) monolith
50 deploys / day
1.5 billion page views / month
http://goo.gl/ZJBiK6
http://goo.gl/XIB29i
@EqualExperts
>600 java microservices
>100 deploys / day
~2 billion requests per day
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
5
Prefer Continuous Improvement over Continuous Delivery
▪ .Net monolith
▪ 26 -> 24 -> 34 releases per year
▪ https://goo.gl/KF9xXw
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
6
People first, practices/process second, products last
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
7
.Net monolith team - how they roll
▪ Private sector
▪ Significant legacy code
▪ High daily traffic (~100 million visits)
▪ ~8 product teams
▪ Weekly deployments:
– Pick release candidate
– Regression test
– Dog food
– Deploy
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
8
Scala microservices team - how they roll
▪ .Gov - HMRC Digital
▪ 2 years old (greenfield)
▪ Calendar spikes (Visits peak is ~4 million)
▪ >30 product teams
▪ Multiple deployments per day
– Teams “own” their own services
– “You build it, you run it” https://goo.gl/DyfeOf
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
9
Patterns for successful practice
Continuous
Delivery
Done Well
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
10
Best practice #1 - Healthy CI - mind & body
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
11
Best practice #1 - Healthy CI - mind & body
Almost
green!
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
12
Best practice #2 - Testing as an activity, not a phase
Product
Owner
Define
@EqualExperts
QA
Dev
Build
Test
@lyndsp
Web Ops
Deploy
© Equal Experts UK Ltd and lyndsayp ltd 2015
13
Best practice #2 - Testing as an activity, not a phase
Product
Owner
Define
QA
Dev
Build
Test
Web Ops
Deploy
Test
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
14
Best practice #3 - Healthy automated tests
UI
API
Unit Tests
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
15
Best practice #3 - (Un)healthy automated tests
The “Hour Glass”
UI
API
Unit Tests
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
16
Best practice #3 - (Un)healthy automated tests
The “Ice Cream Cone”
UI
API
Unit
Test
s
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
17
Best practice #3 - Healthy automated tests
The “Tear Drop”
UI
API
Unit Tests
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
18
Best practice #4 - Low (manual) cost
deployment (and rollback)
VS.
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
19
Best practice #5 - Metrics, monitoring and alerting
“Big data”
Social media
Monitor
all things!
User behaviour
Business metrics
Service performance
Service errors
Disk and Network
http://seluxkanaur.deviantart.com/art/Sauron-s-Eye-6-420535318
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
20
Metrics, monitoring and alerting - getting it right
Time & Pain increasing!
Detect
Alert
Respond
Display
Analyse
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
21
Metrics, monitoring and alerting - getting it wrong
Time & Pain increasing!
Detect
Alert
Respond
Display
Analyse
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
22
Metrics, monitoring and alerting - where and how?
Detect
Alert
Display
@EqualExperts
@lyndsp
Respond
Analyse
© Equal Experts UK Ltd and lyndsayp ltd 2015
23
Metrics, monitoring and alerting - stories from the trenches
Detect
Alert
Display
@EqualExperts
@lyndsp
Respond
Analyse
© Equal Experts UK Ltd and lyndsayp ltd 2015
24
Best practice summary
Product
Owner
1. Healthy CI (mind and body)
Define
2. Testing as an activty, not a phase
Build
Web Ops
QA
Dev
Deplo
y
Test
U
I
Test
3. Tear drop shape automated tests (focus on behaviour, with loose coupling)
API
Unit
Tests
4. Low (manual) cost deployment and rollback
5. Metrics, monitoring and reporting
Detect
Alert
Display
@EqualExperts
@lyndsp
Respond
Analyse
© Equal Experts UK Ltd and lyndsayp ltd 2015
25
Shared Pain
Continuous
Delivery
Done Well
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
26
Shared pain #1 - Zero down-time releases
https://en.wikipedia.org/wiki/VO2_max
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
27
Shared pain #2 - Manual deployment steps
VS.
https://en.wikipedia.org/wiki/Tailor
@EqualExperts
https://en.wikipedia.org/wiki/Zara_(retailer)
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
28
Shared pain summary
1. Zero down time releases
2. Manual deployment steps
VS.
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
29
Team specific accelerators
Continuous
Delivery
Done Well
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
30
Accelerators - 5 why’s
● Create a timeline.
● Gather those involved.
● Walk through the timeline.
● Choose an entry point event:
○ Event
○ Why?
○ Process improvement
● Repeat x5 (ish)
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
31
Accelerators - 5 why’s - example
Event
Why?
Improvement
Error threshold alert
triggered.
Calls to service X were
timing out.
Implement failover / circuit
breaker.
Calls to service X were
timing out.
Queries run by Service X
were taking too long.
Alert on long running
queries.
Queries run by Service X
were taking too long.
Index was missing for a
new field.
...
...
...
...
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
32
Accelerators - automated release notes - the problem #1
Much confusion
and pain
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
33
Accelerators - automated release notes - the problem #2
Payments
Features
Agile Project
Management
Tool
API’s
Source control
system
Login
Stories
Reporting
Defects
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
34
Accelerators - automated release notes - the solution
1. Identify source changes
Features
Payments
API’s
Agile Project
Source control
Management
system
Tool
Stories
Login
Reporting
2. Identify stories/defects
3. Identify areas
4. Group by story/defect
5. For each, summarise areas
and authors
6. Group by:
Defects
a. What’s done
@EqualExperts
@lyndsp
b. What’s in progress
© Equal Experts UK Ltd and lyndsayp ltd 2015
35
Team specific accelerators
Continuous
Delivery
Done Well
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
36
Accelerators - teams “own” their own microservices
@EqualExperts
▪
“You build it, you run it” - https://goo.gl/DyfeOf
▪
Prefer service over library dependencies
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
37
Accelerators - backward compatible API’s
(principle of least surprise)
Production
Everyone
shares
(and cares)
Staging
QA
Dev
https://en.wikipedia.org/wiki/VO2_max
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
38
Team specific accelerators summary
Monolith:
1. 5 -why’s
Payments
Features
API’s
Agile Project
Management
Tool
2. Automated release note
Microservices:
Source
control
system
Login
Stories
Reporting
1. Microservice dependencies, not library dependencies
Defects
2. Backward compatible API’s
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
39
References and resources
▪ Build Quality In (promo code - LP2015 - https://goo.gl/ZRuQKw)
Implementing Lean Software Development
Continuous Delivery
Release It
@EqualExperts
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
40
Thank
You
Thank You
UNITED KINGDOM
INDIA
CANADA
PORTUGAL
+44 203 603 7830
+91 20 6607 7763
+1 403 775 4861
+
helloUK@equalexperts.com
helloIndia@equalexperts.com
helloCanada@equalexperts.com
helloPortugal@equalexperts.com
Equal Experts UK Ltd
Equal Experts India Private Ltd
Equal Experts Devices Inc
Equal Experts Portugal
30 Brock Street
Office No. 4-C
205 - 279 Midpark way S.E.
Rua Tomás da Fonseca
London NW1 3FG
Cerebrum IT Park No. B3
T2X 1M2
- Torres de Lisboa
Kumar City, Kalyani Nagar
Calgary, Alberta
Torre G, 5º Andar
Pune, 411006
LinkedIn
linkedin.com/company/equal-experts
uk.linkedin.com/in/lyndsp
@EqualExperts
1600-209 Lisboa
Making Software. Better.
Web
Twitter
www.equalexperts.com
@EqualExperts
simple software
solutions to big business problems.
www.lyndsayp.com
@lyndsp
© Equal Experts UK Ltd and lyndsayp ltd 2015
@lyndsp