Death to Manual Deployments
Implementing Consistent, Rapid Releases
Lance Smith
DevOps Tech Rep, IBM
Credit to Eric Minick, IBM
Please note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment,
promise, or legal obligation to deliver any material, code or functionality.
Information about potential future products may not be incorporated into any
contract. The development, release, and timing of any future features or
functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM
benchmarks in a controlled environment. The actual throughput or performance
that any user will experience will vary depending upon many factors, including
considerations such as the amount of multiprogramming in the user’s job stream,
the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results
similar to those stated here.
Confession: I hate manual deployments
•
•
•
•
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
Why do people like manual deployments?
• I feel more in control from the command line
• It’s how I’ve always done it
• I know what I did and see the output of the commands I ran
• When something goes wrong, I can be the hero and fix it
?
Scripts to the rescue?!?
• I don’t have a manual deployment: I have scripts!
–
–
–
–
–
How many scripts?
What order?
Who has permissions to run them?
Are they the same across environments?
What else do you need to do?
• Scripts are only part of a fully automated deployment.
I love automated deployments
•
•
•
•
They’re NOT slow
They ARE consistent across environments
You DON’T fail in production
You DO know who deployed what, where or when
Automated deployments provide benefits
• Provide a platform you can build on
– continuous deploys, smoke tests, other goodies
• Empower people while increasing control
– stop waiting on other people
– stop being harassed by other people
• Better, Faster, Cheaper (Pick any 3)
The Program
• Today’s Manual Deployments
• Elements of an Automated Solution
• Best Practices with Automated Deployments
The Program
• Today’s Manual Deployments
• Elements of an Automated Solution
• Best Practices with Automated Deployments
… you might have a manual deployment
•
•
•
•
•
•
Hand-offs are through email (or file shares)
Developers, QA, and Ops deploy differently
Tweaks communicated on an as-broken basis
Deployments steps are a huge document
The term “deployment weekend” is used
Developers have production passwords
Hand-offs are through email (or file shares)
•
•
•
•
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
?
Developers, QA, and Ops deploy differently
•
•
•
•
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
Tweaks communicated as-broken
•
•
•
•
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
The main page won’t load?
Oh… Right, you needed to
add another data source.
Deployments steps are a huge document
•
•
•
•
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
The term “deployment weekend” is used
• They’re slow
• They aren’t consistent across
environments
• You fail in production
• You don’t know who deployed what,
where or when
That’s ok, I
don’t really
like my kids
anyway
Developers have production passwords
•
•
•
•
They’re slow
They aren’t consistent across environments
You fail in production
You don’t know who deployed what, where or when
♫
If only people were more disciplined…
• Deployment docs
– could capture everything
– always be updated
– could be followed them, perfectly, every time.
• Audit spreadsheets
– could capture everything
– could update them
Image from: http://www.ultimatebasictraining.com/blog/2010/02/what-area-drill-sergeants-like/
People aren’t machines
Creative
Consistent
Automated deployments leverage the strengths of
people and machines.
Alistair Cockburn, “Characterizing People as Non-Linear, First-Order
Components in Software Development”
Automated deploy dev to prod isn’t natural
Conway’s Law: organizations are constrained to produce
solutions that copy the structure of their organization
Automated deploy dev to prod isn’t natural
Conway’s Law: organizations are constrained to produce solutions
that copy the structure of their organization
• Ops wants to maintain control
• Development wants fast feedback
• QA lacks influence in DEV & PROD
The Program
• Today’s Manual Deployments
• Elements of an Automated Solution
• Best Practices with Automated Deployments
Elements of an automated solution
•
•
•
•
•
•
Automated process
Coordinating tiers
Environment definitions
Artifact repository
Access control
Audit trail
Automated process
Fast, consistent and repeatable
Status Quo
• Don’t exist
• Incomplete
• Multiple order-dependent
scripts
• Multiple versions per
environment (or machine)
• Uncontrolled
What we need
• Single script reused
across environments
• Controlled access to
deployment steps
• Auto record what
happened
Coordinating Tiers / Components
Move all tiers, services, config etc together
Status Quo
What we need
• Based on oral history
• In a spreadsheet
• Plan is redesigned every
time based on changes
• A model for whole
systems
• Standard deployment
based on changes to all
pieces and parts
• Actual deployment runs
the needed subset.
Snapshots
Transitions of Components
Dev
QA
Prod
?
?
Transitions of a Snapshot
Dev
QA
Snapshot
Snapshot
Prod
Environment definitions
Per-environment configuration and passwords
Status Quo
What we need
• Configured manually
• Details embedded in
binaries, scripts, property
files, ini files
• Environment information
either uncontrolled or in
source control
• Environments a first-level
entity
• Properties controlled at the
environment level
• Secure storage of sensitive
values (passwords)
Artifact repository
Know what you’re getting
Status Quo
What we need
•
•
•
•
•
• Definitive Software Library
• Consistent set of artifacts
across environments
• Tamper-proof artifacts
• Automated retention policies
based on deployments
Is email a repository
File share on the network
Source control for binary control
Uncontrolled
Crude control over what to keep
Access control
Who can deploy to which environment
Status Quo
• Controlled by who can log
onto the machine
• Access tends to grow over
time in response to crisis
• Access control distributed
across machines
What we need
• Role-based access control
• Access control by
environment
• Single source of access
control
• LDAP / Active Directory
authentication (or
authorization)
Audit trail
Know what happened
Status Quo
What we need
• What audit trail?
• I have the spreadsheet
• Our Change Management
System has what we think
we did
• Audit trail starts at PROD
• Win at Clue:
–
–
–
–
Who
What
Where
When / How (better than
Clue!)
• Require no additional work
• End-to-end traceability
Solution Benefits
• We have confidence that our production deployments will
succeed
• QA can take new builds faster
• DEV deployments fast enough to keep up with our rate of
change
• We have an audit trail from day one
• We have separation of duties
The Program
• Today’s Manual Deployments
• Elements of an Automated Solution
• Best Practices with Automated Deployments
Automated Deployments Best Practices
•
•
•
•
•
•
Deferred deployments
Self-service deployments
Automatic deployment
Automatic smoke-test
Automated rollbacks
Continuous production deployment
Deferred deployments
• Story: I want to do my deployment at 11 pm on Friday after my
users are offline.
• Solution: Schedule deployment for desired time, go home. Wait
for notification of success or failure.
Self-service deployments
• Story: QA wants to be able to take a new build when they need
one.
• Solution: Use per-environment access control to allow the QA
people to deploy on-demand, but only to the test environment.
Automatic deployment
• Story: Want to keep the development test server in sync with the
latest changes in source control.
• Solution: After building, automatically deploy to the development
server.
Automatic smoke-test
• Story: I want to know if something breaks immediately.
• Solution: After deploying, automatically trigger a suite of
automated tests and report the results.
Automated rollback
• Story: If my deployment fails I want to restore the previous build.
• Solution: When a deployment step fails, trigger the restoration of
the last successfully deployed build.
Continuous production deployments
• Story: When a change is committed it should be deployed to
production if it doesn’t break anything.
• Solution: After deployment to test environment run automated
test suites. If they all pass, trigger a deployment to production,
then test that production deploy was successful. Rollback on
failure.
Summary
• Manual deployments are slow and broken because they require
people to act like machines
• Computers make good machines
• Automated deployments provide a solid foundation for further
improvement
Deployment and Release Products
• UrbanCode Deploy
– Application Release
Automation
• UrbanCode Release
– Release management
planning and
release weekend execution
Great Resources for more learning!
 IBM’s DevOps Developerworks Community
 IBM Public DevOps Page:
 DevOps For Dummies Book:
 Release and Deploy For Dummies Book:
41
41
 Slideshare.net/Urbancode
 Ibmdw.net/urbancode/documents
 Ibmdw.net/urbancode/blog/
 lansmith@us.ibm.com
42
Thank You!