Continuous Delivery Implementation
Internal adoption and DevOps proceses and tools within enterprise projects
Robbie Minshall
IBM Software, Rational
June 20 2013
© 2013 IBM Corporation
Agenda
https://jazz.net/wiki/bin/view/Main/ContinuousDeliveryImplementationMeetings
Purpose:
Describe and discuss the scope, initial thoughts and working environment for Continuous
Delivery Implementation
Agenda:
5 min: [Frank] Welcome and why this is important
15 min: [Robbie] Project overview and strategy
25 min: [All] Discussion. Open for all to state a perspective, describe problems or opportunities
that they are excited about. 5 min limit per topic.
10 min: [Robbie] Next steps
2
© 2013 IBM Corporation
3
Legend:
*: Contractor
^: Co-op
~: Supplemental
Dibbe Edwards
Rational Development VP
RTP, USA
???
Bulid & Install Manager
RTP, USA
Frank Varone
Manager Continuous
Delivery Infrastructure
RTP, USA
Xue Ming Zuo (Eric)
Dev Mgr; Workload Patterns
(Beijing, China)
Martha DasSarma – Build Lead
keeping the clm builds running
India (PfMgr: Pradeep Balachandran)
Divya Vidyadharan – Build
Robbie Minshall – Chief Architect
Mary Yost – Project Manager
Kevin Williams – Cloud/pattern Architect
Brad Herrin – Pattern / Automation
Jennifer Liu – Pattern / Automation
John Gambino^ - Automation
UK (PfMgr: Paul Strachan)
Kenneth Thomson – Pattern / Automation
Canada (PfMgr: Rodolfo Napoles)
Tanya Wolff – Pattern / Automation
Maneesh Mehra - Pipeline Automation lead
Thomas Picoli - SSE / Pipeline Automation
Thomas Neal - Pipeline
Christophe Elk – JAZZ IHD
Extended Team Members:
Jason Vasilew – IWD/ SCP / TSAM
India (PfMgr: Krishna Kishore)
Sanket Jain – Build
Canada (PfMgr: Judy Keen)
Li Zhu – Build
Mexico (PfMgr: Guadalupe Torres)
Dainer Zavala Priego – Build
Chris Daly – Install Lead
Corinna Schultz – Install
Zhi Cheng Liu
Xiang Yue Gao
Xue Po Wang
Ting Hui TH LI
TBH 2 in July
TBH 3 to do next version work
Yi Ming W Wang^
Sheng Rong SR^
UK (PfMgr: Paul Strachan)
Mark Anderson – Install
Alan Forbes – Build
Extended Team Members:
Rajeev Sikka – Install
Allen Zhou – Install
Christopher Maguire - Build
Peter Steinfeld - Build Architect
Chad D Montgomery - new open source prototype (Maven based)
- new build tooling strategy …
Nathan Bak – Jazz Platform
Guadalupe Torres – non clm builds
© 2013 IBM Corporation
Market shifts are fundamentally changing the way businesses
approach software driven innovation
Macro Business Environment
Empowered Users
Consumerization of IT drives
the need for apps with high
quality customer experience
Increasing regulatory environments drive
the need to reduce risk
Multi-sourcing
Increasing outsourcing drive
the need for governance
Big Data
Cloud
Social
Mobile
Technology Trends
Intelligent/
Connected Systems
Mobile, social, big data, cloud,
intelligent/connected systems
drive the need for agility
1. Clients are trying to rapidly innovate and become agile with systems of engagement
At the same time need to evolve their systems of records and manage regulatory compliance,
reduce costs by outsourcing and provide world class customer experience
2. The above factors have resulted in shift in purchase dynamics
Developers influencing LOB buyers for entire delivery stack (cloud, mobile, middleware, tools)
4
© 2013 IBM Corporation
Organizations that effectively leverage software innovation
outperform their competitors... yet few are able to deliver it effectively
86
%
of companies believe software delivery
is important or critical
But only…
25
%
of those who
leverage software
delivery today
69
%
outperform
those who don’t
leverage software delivery effectively today
5
Source: “The Software Edge: How effective software development drives competitive advantage,” IBM Institute of Business Value, March 2013
© 2013 IBM Corporation
Why is this important
 This is really really hard
 Continuous Integration, Test and Delivery of Software within non trivial landscape
 CLM Relevance
– 4th Major release
– Significant Legacy
– Historically this is the point where products struggle to delivery incremental value and innovation
 Rational
– Customers recognize that innovation is essential to their business and that software is the important part
of that (first time that it has ranked top of the list)
– Customers are looking for ways to increase speed of software innovation, the developer is king
– Rational should be as well positioned in this space. We are not.
6
© 2013 IBM Corporation
Executive Perspective
Rational Continuous Delivery adoption
2008
2010
2012 – 2013
Total Improvement
Project Initiation
30 days
10 days
2 days
28 days
Groomed Backlog
90 days
45 days
On-going
89 days
Overall TTD
120 days
55 days
3 days
117 days
Iteration Length
6 weeks
4 weeks
4 weeks
4 weeks
6
8
3
N/A
36 hours
12 hours
8 hours
400 %
N/A
18 hours
< 1hour
17 hours
Iteration Test Time
5 days
2 days
4 hours
4 days
Total Deployment Time
2 days
8 hours
2 hours
2 days
Overall TTP
9 days
3 days
15 hours
8 days
12 Months
12 Months
3 Months
9 Months
Lifecycle Measurements
Number of Iterations
Composite Build Time
BVT Availability
Time Between Releases
Consume and
Drive
Processes and
Culture
External
Deliverables
Measure and
Optimize Value
DevOps Transformation Project
Drive
adoption
© 2013 IBM Corporation
Continuous Delivery Example Maturity Model
Industry norm
Level 1
Level 2
Target attainment
Continuous
Business Planning
Collaborative
Development
Continuous
Testing
Continuous
Release and
Deployment
Continuous
Monitoring
8
8
Level 4
Level 5
• Real-time dashboard of
business value
attainment
• Integration of business
and customer metrics
• Full implementation of
lean principles
• Real-time integration
entire lifecycle spanning
from business objectives
to customer sentiments
• Business objectives
documented in silo
• Business objectives
linked to development
work items
• Ability to objectively trade
off requirements
• Visibility to development
status
• Ad-hoc build scripts
• Siloed developers
• Bare-bone SCM (e.g.
non-atomic commits)
• Nightly builds
• Usage of methodologies
(Agile, Scrum…)
• Basic SCM (basic
tagging, merging)
• Build on commit
• Publish to asset
repository
• Real-time work item
planning
• Build verification tests on
production-like
environments
• Advanced SCM (efficient
personal branch/merge)
• Lifecycle traceability
• Development intelligence
• Continuous process
improvement
• Use quality management
tools
• Continuous integration
testing w/service
virtualization
• Static code analysis
• Security testing
• Coverage analysis
• Risk-based testing
• Fully automatic
acceptance tests
• Ad-hoc deployment
scripts
• Self-service deploy to
test environments
• Mostly standardized
deployments
• Self-service deployment
to test and production
environments
• Fully standardardized
deployments
• Quality-gated automatic
promotions
• Deployment of all
application artifacts (e.g.
database updates)
• Coordinated SOA/multitier deployments
• Continuous deployment
to production
• Reactive performance
resolution (e.g. outage
alerts)
• Performance
instrumentation and
monitoring
• Real-time performance
dashboards
• Automated performance
tests as part of quality
gates
• Continuous performance
optimization
• Automatic crash reporting
• User session recording
and reporting
• In-app customer
feedback
• Log analysis
• Measuring customer
experience KPIs
• A/B, Split and
Multivariate testing
• Advanced business
impact analysis
• Big-data analysis
• Voice of the customer
investigation
• Self-evolving application
based on A/B testing
• Some test automation
Will be updated!!!!!
Continuous
Customer
Feedback and
Optimization
Level 3
© 2013 IBM Corporation
Back to the future: WAS Adoption of WCA
Background
Background and Challenges
Shorten time delivery time via Agile Processes
Provide meaningful dev & test environments
Increase Hardware Utilization
Decrease system admin costs
WCA Solution
Construct Private Cloud leveraging WCA
Seed Cloud with 6% of Infrastructure
Adopt and scale to 50% over 4 years
Realized Results
Realized Benefits
Deployment reduced from 3+ hours to 20 minutes
Hardware Utilization increased from 6% to 60%
User Deployment failures down from 50% to 5%
Complete Performance Environment as Patterns
Increased system admin efficiency 16 fold
Realized Results 1Q 2009 – 2Q2010
355 K in Labor efficiency gains
160 K in Infrastructure savings
Enabled Results due to Process Changes
2.1 million Labor efficiency gains
780 K / year downstream regression test savings
Projected Results
Projected CloudBurst Adoption Rate :
2010 = 6%,2011=20%,2012=40%,2013/14=50%
Benefit Summary:
Setup = 24K
Training = 36K
5 year return on investment = 10.4 Million
© 2013 IBM Corporation
Interact
Back to the future
WebSphere deploys private cloud driving Agile Adoption and
gain efficiencies throughout delivery
Client Pains
 Enable rapid access to complete testing
topologies to improve quality and
decrease costs
 Drive down costs and increase utilization
of infrastructure
Real Results
 1,700 K in direct savings, 2.1 Million/year in enabled efficiency.
 Reduced topology install time from 3 Hours to 20 Minutes.
Increased hardware utilization by 54% . Increased System Admin
efficiency by 1600% compared with bare metal.
© 2013 IBM Corporation
Back to the future: Recipes for success and lessons
 Improve developer experience and functional stability
 Realize internal efficiencies
 Adopt and drive IBM DevOps technologies
 Produce a functioning DevOps Reference Implementation
 Contribute to IBM thought leadership
© 2013 IBM Corporation
Desired Customer Perspective for Transformation Effort
To be honest we either never quite bought in, or were never successful with Agile Development. Frankly, while our development teams
may have been agile that did not really translate as we had hoped into faster application delivery. We really struggled with how to engage
our operations team and to meet the speed requirements for social and mobile applications. There was always this tension between the
development of those next generation applications and how that related to our traditional systems. Once you introduce public and private
cloud technologies into the equation we were lost and were looking for a model based on real results. I was sick of hearing the words Agile,
DevOps and Cloud, help me know how to realize some results.
The Rational DevOps Transformation project helped us make sense of this mess and I loved the transparency. Think of it as a DevOps
Reference Architecture, Adoption Path, Business Value Assessment and Deployable solution that not only tells the story of IBMs
transformation but provided a blueprint for ours.
It was first brought to our attention by our Executives who pretty much wanted to duplicate the results. We were skeptical but their transition
story was transparent and helped us understand a set of best practices and lessons.
One of the key takeaways was that we needed to invest in this transition. Ultimately there was of course exciting new technologies to adopt
but also changes in our organization and lines of business were necessary. The business value assessments created a model that
helped us with these adoption challenges.
From a technology standpoint we were able to pilot DevOps solution very quickly because we were able to take what IBM used internally
and deploy it quickly in order to demonstrate the value we were looking for.
From there it was a matter of integrating the functioning solution with out existing systems and creating a strategy for realizing value in our
Systems of Record as well as our emerging applications.
If you are looking for validation that DevOps is achievable based upon real results in a non trivial environment then check out the DevOps
Transformation Project and IBM DevOps technologies.
© 2013 IBM Corporation
~ 1 week
Developer Interaction
Production
Deployable Solution on Private Cloud
Draft Pipeline
Smarter Pipes Service
Manual Tests
Release
Dashboards
Production like
Development
uBuild
Jazz.net
Production
Production like
System Test
App Scan
Unit Test
Integration Test
Repository
Rational
Collaborative
Application
Lifecycle
Management
Function Test
Performance
Test
Integration Test
JazzHub
Production
Usability
Consume and
Drive
Processes and
Culture
External
Deliverables
Measure and
Optimize Value
Drive
adoption
DevOps Transformation Project
© 2013 IBM Corporation
Project Structure
14
© 2013 IBM Corporation
Emerging timeline
15
© 2013 IBM Corporation
Ways of working
 T skills
– Deep technical knowledge
– Broad awareness and contribution
 Deliver
– 90% of what YOU do should have a known customer or developer deliverable when you start
 Collaborate
– Engage the organization
 Win
– We will be measured by the developer experience
16
© 2013 IBM Corporation
Discussion
5 minute topics
Open for all to state a perspective, describe problems or
opportunities that they are excited about. 5 min limit per topic.
No Whining
17
© 2013 IBM Corporation
Discussion Point: Smart Pipes
 Problem: static test pipe of increasing complexity/value is too slow for continuous integration
 Constantly quantify the value of a test case in terms of it's reliability, it's history of finding
defects, the variety of systems that it finds defects in, and whether or not it is related to
change sets. For various build/integration purposes have a defined regression window.
Within that window dynamically organize the pipeline in the most optimal way to ensure the
driver is stable, to find regressions as quickly as possible and to begin expensive or time
consuming tests cases such as long runs earlier enough to matter.
 Said another way run the best tests to ensure stability, don't run stuff that never matters, run
the most relevant tests, run stuff concurrently not linearly, start valuable long running tests as
soon as stability is determined. Do all of this within a relevant execution window and provide
data and an assurance metric throughout the process.
 A smarter pipeline would become an important part of a best of breed Application Release
System that is necessary to differentiate IBM from open source ALM and emerging devOps
solutions.
 Question: With what | Who | When
Consume and
Drive
Processes and
Culture
External
Deliverables
Measure and
Optimize Value
Drive
adoption
DevOps Transformation Project
18
© 2013 IBM Corporation
Discussion Point: Delivery mechanisms to Production
 Jazz.net
 JazzHub
 Internal adopters (Hursley, and Rational)
 Workload Patterns
 DevOps developerWorks portal
 Traditional Deploy Repositories
Consume and
Drive
Processes and
Culture
External
Deliverables
Measure and
Optimize Value
Drive
adoption
DevOps Transformation Project
19
© 2013 IBM Corporation
Discussion Point: Applying service
 How should upgrades and service be applied in JazzHub and Jazz.net environments?
– Swim-lanes / Hold Cold
– Patch
– Rolling upgrades
 How are the same mechanisms used within the our delivery pipeline
 Do all new project requests get placed on latest available (JazzHub)
 How many topology deployments at a time (JazzHub)
Consume and
Drive
Processes and
Culture
External
Deliverables
Measure and
Optimize Value
Drive
adoption
DevOps Transformation Project
20
© 2013 IBM Corporation
Next Steps
What are the next steps
21
© 2013 IBM Corporation
Next Steps
 June – July (Learning, evaluation and communicating a perspective)
 Form mini projects with dedicated resources
 Create some initial horizontal teams
 Merge various efforts and understand what is not going to be done
 Establish baseline business value assessments
22
© 2013 IBM Corporation