Highway to heaven
Building microservices in the cloud
Frankfurter Entwicklertag 2016
Agenda
Meet AutoScout24
Shifting gears
How we build our services
How we organize ourselves
@cdeger & @wolfwolf
AutoScout24 - 30.000ft
C
C
C
Baseline AutoScout24 IT
Highly optimized, but of last decade
IT platform supported growth for >6 years
Microsoft oriented stack
Enterprise IT setup - MTBF over MTTR
Proven agile and lean principles
C
C
New CEO
C
C
Do you attract talent?
C
C
C
C
Project Tatsu
W
W
Five challenges
Monolith to Microservices
Data center to AWS
.NET / Windows to JVM / Linux
Devs + Ops to DevOps culture
Involve product people
Technical transformation
W
W
Change the wheels while driving
W
W
Strategic Goals
Architectural Principles
Design and Delivery Principles
Goals of the business side
High-Level Principles
Tactical measures
Reduce Time to Market
Organized around Business Capabilities
You build it, you run it
Speed, Fast Feedback
Build teams around products not projects. Follow the domain
and respect bounded contexts. Inverse Conway Maneuver.
The team is responsible for shaping, building, running and
maintaining its products. Fast feedback from live and customers
helps us to continuously improve.
Support Data-Driven Decisions
Containment and Boundaries
Listen to users and validate hypothesis.
Provide as many relevant metrics & data as possible.
Align blast radius and vendor lock-in with the boundaries of the
organization or business capabilities.
Cost Efficiency
Eliminate Accidental Complexity
Collect metrics to allow decisions cost vs. value.
Strive to keep it simple. Focus on essential complexity. You build
one, you delete one.
Another goal
Shared Nothing
Work in progress...
By default avoid sharing and tight coupling, except for the big
things in common. Don’t create the next monolith.
Macro and Micro Architecture
Clear separation. Autonomous micro services within the rules
and constraints of the macro architecture.
Security, Compliance and Data Privacy
Security must be included from the beginning and everybody’s
concern. Keep data-privacy in mind.
Collaboration Culture
Engineers from all backgrounds work together in collaborative
teams as engineers and share responsibilities. No silos.
Autonomous Teams
Make fast local decisions. Be responsible. Know your
boundaries. Share findings.
Be Bold
Go into production early. Value monitoring over tests. Recover
and learn. Optimize for MTTR not MTBF.
Data-Driven/ Metric-Driven
Collect metrics from processes and applications. Analyze, alert
and act on them.
Infrastructure As Code
Automate everything: Reproducible, traceable and tested.
Immutable servers over snowflake servers.
AWS First
C
Favor AWS platform service over managed service, over selfhosted OSS, over self-rolled solutions.
C
Principles
Organized around business capabilities
You build it, you run it
Be bold
Macro and micro architecture
Shared nothing
Shared infrastructure
Shared nothing
Availability over shared nothing
Convenience offerings
No side effects
Fast local decisions over committee
W
Respect family ties
W
How (not) to share
Use over re-use
Re-use only after hardening
Copy n’paste, OSS, library
Pull instead of push
EventPublisher24
W
W
How many environments?
Engineer
CI
Dev
Staging
Prod
V5
V4
V3
V2
V1
V8
V7
V6
V5
V4
Which versions on staging?
Prod differs anyway: Load, data, patterns
C
C
Dev and prod is enough
Engineer
CI
Dev
Prod
V5
V4
V3
V2
V8
V7
V6
V5
Consumer driven contracts
Shadow traffic
Semantic monitoring
Smoke tests or canary releases
C
C
Event Sourcing and data pumps
One way data highway
Event Sourcing - store history of all changes
W
W
Evolution
SQS + S3
Kinesis + S3
Kinesis + DynamoDB
SQS + DynamoDB
Proxy + DynamoDB
W
DynamoDB
W
Frontend integration
Shared nothing
Autonomous teams
One domain
High optimisation
C
C
Frontend integration
C
C
PageSpeed Module
css (page)
css (page+fragment)
css (fragment)
ngx_pagespeed
js (fragment)
js (page)
C
js (page+fragment)
C
Wolf
How to build autonomous teams
Do not fall back into old behaviours
Beware of Mandelbrot teams
Pager duty so that you run it
Fix broken windows
Part-time ops not working
Not all T-shapes are the same
W
W
Infrastructure guild
Agree on things to do
Share learnings
Delegate implementation to teams
Empty backlog should be normal
Infrastructure product teams needed?
W
W
cdeger@autoscout24.com @cdeger
wschlegel@thoughtworks.com @wolfwolf
Attributions
Blue sky, white-gray clouds by nature protector Natubico, www.vivism.info [CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3ABlue_sky%2C_white-gray_clouds.JPG
A Danish Perspective by NASA [Public domain] http://commons.wikimedia.org/wiki/File%3AA_Danish_Perspective.jpg
http://commons.wikimedia.org/wiki/File%3ANASAComputerRoom7090.NARA.jpg
GREG
EINRAD
Amazon16 by Neil Palmer/CIAT [CC BY-SA 2.0] https://www.flickr.com/photos/ciat/5641594952
BERGSTEIGER
Barber in Cameroon by James Emery from Douglasville, United States (Daddy Joe_1355) [CC BY 2.0]
http://commons.wikimedia.org/wiki/File%3ABarber_in_Cameroon.jpg
Wide objectives by Kivela (Own work) [Public domain]
href="http://commons.wikimedia.org/wiki/File%3AWide_objectives.jpg
Transformer Fire Barrier by GerryS1 (Own work) [CC BY-SA 3.0 or GFDL]
http://commons.wikimedia.org/wiki/File%3ATransformer_Fire_Barrier.jpg
Attributions (cont)
Alonso Renault Pitstop Chinese GP 2008 by Bert van Dijk (Pitstop F1 ING Renault) [CC BY-SA 2.0]
http://commons.wikimedia.org/wiki/File%3AAlonso_Renault_Pitstop_Chinese_GP_2008.jpg
Principle of Panchasheel by Prakash Adhikary (Own work) [CC BY 3.0]
http://commons.wikimedia.org/wiki/File%3APrinciple_of_Panchasheel.JPG
Traffic Jam by Doo Ho Kim [CC BY-SA 2.0] https://www.flickr.com/photos/titicat/3049591547
Pellets by The original uploader was Richard Mayer at German Wikipedia [GFDL or CC-BY-SA-3.0]
http://commons.wikimedia.org/wiki/File%3APellets.jpg
Pipes and Valves by Uwe Hermann [CC BY-SA 2.0] https://www.flickr.com/photos/73628542@N00/6272975359
Size variation in Coccinella undecimpunctata (2127991716) by Gilles San Martin from Namur, Belgium [CC BY-SA 2.0]
http://commons.wikimedia.org/wiki/File%3ASize_variation_in_Coccinella_undecimpunctata_(2127991716).jpg
Mille crêpe by Laitr Keiows (Own work) [CC BY-SA 3.0 or GFDL]
http://commons.wikimedia.org/wiki/File%3AMille_cr%C3%AApe.jpg
Country Energy power line replacement 01 by Bidgee (Own work) [CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3ACountry_Energy_power_line_replacement_01.jpg
Puzzling by Bernd Gessler (Own work) [CC BY-SA 3.0] https://commons.wikimedia.org/wiki/File%3APuzzling.JPG
Attributions (cont)
Sharing Sucks (4536747557) by eyeliam from Portland, United States [CC BY 2.0]
http://commons.wikimedia.org/wiki/File%3ASharing_Sucks_(4536747557).jpg
7Line 9184 (8263568241) by Metropolitan Transportation Authority of the State of New York (7Line_9184 Uploaded by
tm) [CC BY 2.0] http://commons.wikimedia.org/wiki/File%3A7Line_9184_(8263568241).jpg
England rugby team 1905 by Russell & Sons (The Graphic) [Public domain or Public domain]
http://commons.wikimedia.org/wiki/File%3AEngland_rugby_team_1905.jpg
Wandergeselle by Sigismund von Dobschütz [CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3AWandergeselle_02.JPG
Faber-Rechenschieber 5304 by User:Karl Gruber (Own work) [CC BY-SA 4.0]
http://commons.wikimedia.org/wiki/File%3AFaber-Rechenschieber_5304.JPG
Wheel clamps Texas by Richard Anderson from Denton, United States (Boots.) [CC BY-SA 2.0]
http://commons.wikimedia.org/wiki/File%3AWheel_clamps_Texas.jpg
GuadalupeNOLA15Oct07Thanks by Infrogmation of New Orleans (Photo by Infrogmation) [GFDL or CC BY-SA 3.0]
http://commons.wikimedia.org/wiki/File%3AGuadalupeNOLA15Oct07Thanks.jpg
AtariBasic by Calin99 (Own work) [GPL] http://commons.wikimedia.org/wiki/File%3AAtariBasic.png
Spare wheel by Brian Snelson [CC BY 2.0] https://commons.wikimedia.org/wiki/File:Spare_wheel_-_Flickr_-_exfordy.jpg
Backup
Unified
Logs
C
C
Two stacks
Cash stack meets shiny new stack
One company
Lights on in cash stack
Feature freeze
Where to build new features?
C
Ease of integration helps business people
C
From documents to events
Refactoring toward deeper insight
From CRUD to sync
Offline first
Writes are expensive
USD 20 over USD 500
C
C
W
How many layers to estimate a price
Evolving architecture
R backend and Play application
Single Play application
Play backend and Play web server
Long feedback cycles
W
Frequency of change
W
C
Watchlist evolution
DynamoDB as JSON document store
Scheduled user journey test
Capacity exceeded
Requests throttled
Latency spikes for real users
C
C
Focus sliders
Product over platform
business
IT
time
W
W
Focus sliders
Product over platform
Delivery over learning
business
IT
time
W
W