A Forrester Consulting
Thought Leadership Paper
Commissioned By Microsoft
DevOps Best Practices
The Path To Better Application
Delivery Results
September 2015
Table Of Contents
Executive Summary ........................................................................................... 1
DevOps Leaders Achieve Faster Delivery And Better Quality ..................... 2
Automation And Integration Are Critical Steps.............................................. 4
Automate Testing To Help Make Data-Driven Release Decisions ............... 5
Smaller Releases Mean Faster Releases ........................................................ 8
Faster Feedback Means Better Results .......................................................... 9
Key Recommendations ................................................................................... 10
Appendix A: Methodology .............................................................................. 11
Appendix B: Supplemental Material .............................................................. 11
Appendix C: Demographics ............................................................................ 12
ABOUT FORRESTER CONSULTING
Forrester Consulting provides independent and objective research-based
consulting to help leaders succeed in their organizations. Ranging in scope from a
short strategy session to custom projects, Forrester’s Consulting services connect
you directly with research analysts who apply expert insight to your specific
business challenges. For more information, visit forrester.com/consulting.
© 2015, Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited.
Information is based on best available resources. Opinions reflect judgment at the time and are subject to
change. Forrester®, Technographics®, Forrester Wave, RoleView, TechRadar, and Total Economic Impact
are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective
companies. For additional information, go to www.forrester.com. [1-TVS6WB]
1
Executive Summary
If you are wondering whether development operations
(DevOps) is merely a fad, you’re already falling behind.
Faster and more nimble competitors are already able to
quickly respond to shifting customer demands to deliver
better customer experiences and better business results.
DevOps provides a proven set of practices that enable
organizations to deliver applications more quickly to better
connect with customers while simultaneously reducing both
cost and risk. The results are profound, but achieving them
requires focus and willingness to leave behind old ways of
working and organizing.
In June 2015, Microsoft commissioned Forrester Consulting
to evaluate application development and delivery best
practices. Then to further explore this trend, Forrester
developed a hypothesis that tested the assertion that the
terms DevOps and continuous delivery (CD) are
synonymous with pipeline automation. For enterprises,
these terms have broader meaning and reflect a journey
toward continuous delivery.
In conducting in-depth surveys with 425 application
developers and IT operations managers, Forrester found
that companies that deliver applications more frequently
(monthly or more) are actively implementing DevOps
practices and seeing measurable results.
KEY FINDINGS
Forrester’s study yielded three key findings:
›
›
DevOps practices enable organizations to increase
satisfaction, revenue, and efficiency. Nearly seven out
of 10 organizations delivering applications monthly or
more frequently expect higher customer satisfaction. On
top of that, 65% of those releasing rapidly anticipate
improvements in efficiency across teams, and 56%
anticipate revenue growth. Faster delivery cycles enable
organizations to provide innovative solutions by quickly
delivering new capabilities and reducing the time they
spend waiting for feedback. They are able to try new
ideas quickly, improve the ones that work, and rapidly
improve or remove the ones that don’t. Better, faster
feedback enables organizations to cut waste, reduce cost,
and improve customer experiences.
DevOps practices build on Agile success to enable
continuous delivery. To push forward on the Agile
journey, application delivery professionals are turning to
DevOps practices like infrastructure as code (IaC),
infrastructure-as-a-service (IaaS) and platform-as-aservice (PaaS) to support application delivery. Other
practices such as continuous integration (CI) are widely
adopted by more than 60% of organizations that deploy
applications monthly or more frequently, compared with
only 48% of those delivering at a slower pace.
›
Agile practices have broadly penetrated most
organizations, but there’s still room to advance. Agile
practices — most specifically, those that deliver
applications in small increments — have widespread
adoption. Approximately 80% of DevOps professionals
work on teams that deliver small increments of value;
however, half of those delivering at least monthly are
adopting widespread incremental delivery. While these
practices improve delivery cycle time, more complete
adoption of DevOps practices leads to even greater
improvements.
2
DevOps Leaders Achieve Faster
Delivery And Better Quality
Increasing competition and customer demand for new
capabilities and better experiences is compelling
organizations to deliver applications faster than ever before.
Mobile and cloud applications are at the center of these
changes, but all application types are affected as
organizations race to win, serve, and retain customers. This
is particularly evident in the way that companies measure
quality: According to our data, half of respondents work for
companies that measure software quality through customer
engagement. In addition, our survey found:
›
›
›
Some organizations deliver applications monthly or
more frequently. Thirty-nine percent of DevOps
professionals reported releasing applications monthly or
more frequently. Another one-third (32%) reported that
they average one release per quarter, while 27% reported
one or two releases per year (see Figure 1). New
application versions are delivered even more frequently,
with updates happening monthly or more frequently in
nearly 80% of companies surveyed.
Agile practices are widely adopted. Respondents have
embraced Agile application development techniques.
When we look more carefully at companies delivering
more frequently, we find that a greater percentage (52%)
are delivering in small increments compared with
companies delivering at a slower pace (46%) (see Figure
2). Agile practices also give organizations an edge in
making release decisions based on test data. Loosely
coupled application architectures, based on services and
microservices, also enable faster application delivery
Simplify processes and teams and automate
wherever possible. Removing constraints and improving
flow leads to faster delivery. Organizations that deliver
more rapidly improve how work flows by removing steps
that add no value and eliminating handoffs and wait time
(see Table 1).
FIGURE 1
Thirty-Nine Percent Release Applications Monthly
Or More Frequently
“How often does your team release
applications?”
Release many times
per day
2%
One release a day
3%
39%
9%
One release a week
25%
One release a month
32%
One release per quarter
21%
Two releases a year
One release a year
(or less frequently)
6%
Base: 425 IT professionals involved in application development and
delivery
(percentages may not total 100 because of rounding)
Source: A commissioned study conducted by Forrester Consulting on
behalf of Microsoft, June 2015
TABLE 1
Eight Steps To Take Action
1. Automate environment management.
2. Integrate continuously.
3. Automate deployments.
4. Automate testing with APIs.
5. Make data-driven release decisions.
6. Reduce size of releases.
7. Eliminate handoffs and wait time.
8. Drive better results with feedback.
Source: Forrester Research, Inc.
3
FIGURE 2
More Frequent Releases Are Driven By Key Practices
“Which best describes the scope of your team’s adoption of each practice?”
(Those who selected “widespread use” of these practices)
Release less frequently (N = 256)
Release monthly or more frequently (N = 168)
50%
Make data-informed release decisions
Deliver in small increments of value
Utilize loosely coupled architectures
60%
46%
52%
42%
47%
Create managed environments that are
available on demand
∆ 10%
∆ 6%
∆ 5%
53%
57%
∆ 4%
Deliver software via an integral, cohesive, self-directing,
multidisciplinary team that spans releases
52%
54%
∆ 2%
Maximize throughput in the delivery pipeline
52%
54%
∆ 2%
Manage and minimize technical debt
Form/test hypotheses about customer needs,
make decisions based on evidence gathered from testing
Prioritize work based on expected business value
Base: 425 IT professionals involved in application development and delivery
Source: A commissioned study conducted by Forrester Consulting on behalf of Microsoft, June 2015
43%
43%
∆ 0%
48%
48%
55%
51%
∆ 0%
∆ -4%
4
Automation And Integration Are
Critical Steps
Agile software delivery practices help organizations
streamline application development, but delivery bottlenecks
can hamper efforts to reach a desired delivery velocity.
DevOps practices that automate environment management,
integrate continuously, and automate deployments increase
delivery speed by systematically removing wait time and
error-prone manual effort, along with the frustrations and
friction they create. Our survey showed:
›
›
›
›
Cloud technologies help organizations deliver faster.
Sixty-three percent of organizations delivering
applications monthly or more frequently use cloud-based
environments, such as infrastructure-as-a-service, to
remove deployment bottlenecks, compared with 56% of
organizations that deliver less frequently. Organizations
that deliver more frequently are also using PaaS more
than those delivering at a slower pace. Fifty-eight percent
of organizations that release monthly or more use
platform-as-a-service, compared with 52% of
organizations that release less frequently (see Figure 3).
Automated delivery pipelines provide consistency in
delivery. More than half (52%) of DevOps professionals
use an automated release pipeline to deploy into
production (see Figure 3). Also, our research found that
40% of companies deploying monthly or more frequently
use their automated release pipeline to establish stages
for progressively harsher testing.
CI is the foundation for the automated delivery
pipeline. Continuous integration provides consistency
during the build, integration, and testing process and
allows teams to detect issues early. According to our data,
CI is used by nearly 60% of those deploying applications
monthly or more frequently, compared with only 48% of
those delivering at a slower pace.
Automating software deployment is also a key
enabler of fast delivery. For two-thirds of organizations,
software is deployed automatically as part of the CI
process. In three out of five companies deploying monthly
or more frequently, software deployment is automated but
performed separately from the CI process, according to
our research. Merging automated deployment with
continuous integration gives organizations an end-to-end
delivery pipeline.
FIGURE 3
Release Techniques In Practice
“Which, if any, of the following release techniques
does your team currently practice?”
Release less frequently (N = 256)
Release monthly or more frequently (N = 168)
55%
Infrastructure as code
Automated release pipeline
deploying into production,
with or without signoff gates
76%
45%
52%
Use of public cloud for
deployment on infrastructureas-a-service (IaaS)
56%
63%
Use of public cloud for
deployment on platformas-a-service (PaaS)
Use of containers
52%
58%
34%
35%
∆ 21%
∆ 7%
∆ 7%
∆ 6%
∆ 1%
Base: 425 IT professionals involved in application development and
delivery
Source: A commissioned study conducted by Forrester Consulting on
behalf of Microsoft, June 2015
Take action:

To start, automate environment
provisioning of development and testing
environments, and then expand to
production environments.

Automate building and testing software
through continuous integration. Start with
basic build and unit test automation, and
then gradually expand automation to
functional, performance, scalability, and
security testing.

Combine release automation with
continuous integration to create a basic
automated application delivery pipeline.
5
frequently (see Figure 4). Standardizing configurations
and automating provisioning not only makes testing
earlier possible, but it eliminates the “it works fine in the
test environment, it will work fine in release” problem.
Automate Testing To Help Make
Data-Driven Release Decisions
Automating testing and incorporating it into continuous
integration from the very beginning provides developers with
timely feedback that they can use to improve application
quality. The resulting test data enables organizations to
make more informed release decisions. Earlier and more
frequent testing is essential in organizations that release
monthly or more frequently. Our study found:
›
›
›
Automating testing environment management
increases release speed and accuracy. Overall, 64% of
respondents reported doing some testing in production
environments, gradually exposing more and more users
as confidence in the release improves. While this limits
the impact of defects in production, it does not provide
feedback early enough to significantly improve quality.
FIGURE 5
How Issues Are Discovered
Standardizing and automating environment
provisioning simplifies and speeds delivery. Among
organizations delivering monthly or more frequently, 52%
reported using infrastructure-as-code techniques,
compared with only 36% of those releasing less
“Thinking about your production environment,
which of the following best describes how
issues are discovered?”
Release less frequently (N = 256)
Release monthly or more frequently (N = 168)
FIGURE 4
Half Of Those Delivering Monthly Use IaC
Techniques
Instrumentation or logging
of application events
triggered by user actions
Automated alerts that are
triggered prior to or by
a disruption in service
“Which of the following methods does your team use
to define environments and deployment
processes for testing?”
Issues are reported by
end users
Release less frequently (N = 256)
Release monthly or more frequently (N = 168)
Testing environments are
created on demand
using infrastructure-ascode techniques
Dedicated individual test
machines (physical or virtual)
Testing is done in production,
with progressive exposure to
users based on successful
test outcomes
Dedicated test labs
with complete virtualized
environments (roles, network)
36%
52%
51%
65%
62%
68%
60%
58%
Instrumenting applications running in production
provide developers with essential feedback. Seventy
percent of DevOps teams delivering more quickly are
relying on instrumentation or logging of application events
triggered by user actions to uncover issues with
applications. These teams are also more likely to rely on
automated alerts, which are triggered before a disruption
in service. On the other hand, organizations delivering
quarterly or less frequently continue to rely on manual
inspections by staffers and end user reports of issues
(see Figure 5).
Manual inspection by staff
∆ 16%
∆ 14%
57%
70%
45%
51%
∆ 13%
∆ 6%
57%
61%
69%
70%
∆ 4%
∆ 1%
Base: 425 IT professionals involved in application development and
delivery
Source: A commissioned study conducted by Forrester Consulting on
behalf of Microsoft, June 2015
∆ 6%
∆ -2%
Base: 425 IT professionals involved in application development and
delivery
Source: A commissioned study conducted by Forrester Consulting on
behalf of Microsoft, June 2015
›
Loosely coupled application architecture simplifies
and speeds delivery. As we saw in Figure 2,
approximately half (47%) of organizations delivering
monthly or more frequently reported using loosely coupled
architectures. Applications composed of relatively
independent services are easier to release and evolve;
their releases require less coordination and are less risky.
Loose coupling also simplifies application development by
6
reducing team dependencies, but it increases the
importance of software architecture to arrive at the right
set of loosely coupled services.
›
Teams that deliver faster manage technical debt more
deliberately. Technical debt, or postponed remedial
technical work, is more closely managed at organizations
delivering faster. More than half (54%) of teams delivering
monthly or more frequently reported that they periodically
launch projects to replace or modernize major legacy
systems, compared with only 45% of those delivering at a
slower pace. Additionally, 52% of organizations with faster
delivery speeds track high-value and/or high-pain
technical debt remediation work, compared with only 36%
of those delivering more slowly (see Figure 6).
Release less frequently (N = 256)
Release monthly or more frequently (N = 168)
36%
“Which, if any, of the following overarching life-cycle
techniques does your team currently practice?”
∆ 16%
45%
Strategically incur short-term
technical debt to enable
experimentation and fast
feedback; we track and
measure this debt and pay
it off promptly
44%
Autonomous,
multidisciplinary teams
Use public-cloud-based
environments for
development and test
52%
Periodically launch projects
to refactor/replace/modernize
major legacy systems
Track and remediate
infrastructure debt along
with application debt
FIGURE 7
Overarching Life-Cycle Techniques In Place
Kanban boards for
portfolio tracking
“Which of the following best describes your
company’s approach toward technical debt?”
Avoid technical debt at
all costs by rigorous
architectural design
and review
Faster delivery demands greater discipline across a
broad set of life-cycle practices. Contrary to common
misconception, teams that deliver faster do not cut
corners to do so. Nearly 60% of those delivering rapidly
reported that they track bugs, work items, and tasks
centrally, and 54% are trending historical usage data (see
Figure 7). In addition, teams delivering more frequently
are also more likely to have autonomous, multidisciplinary
teams and a common definition of “done.”
Release less frequently (N = 256)
Release monthly or more frequently (N = 168)
FIGURE 6
Managing Technical Debt
List high-value and/or
high-pain technical debt
remediation work in
our backlog
›
Common definition
of “done”
Kanban boards for
project item tracking
∆ 9%
54%
∆ 8%
Trend historical usage
data for improvement
opportunity
Common product backlog
23%
∆ 13%
36%
32%
43%
40%
51%
29%
38%
∆ 11%
∆ 11%
∆ 9%
27%
35%
∆ 8%
46%
54%
35%
43%
∆ 8%
∆ 8%
52%
Track bugs, work items,
and/or tasks centrally
38%
∆ 1%
39%
46%
∆ 0%
46%
Base: 425 IT professionals involved in application development and
delivery
Source: A commissioned study conducted by Forrester Consulting on
behalf of Microsoft, June 2015
Use private cloud
environments for
development and test
Trend historical
production data for
improvement opportunity
Trend historical
development data for
improvement opportunity
51%
58%
∆ 7%
41%
47%
∆ 6%
46%
48%
∆ 2%
44%
46%
∆ 2%
Base: 425 IT professionals involved in application development and
delivery
Source: A commissioned study conducted by Forrester Consulting on
behalf of Microsoft, June 2015
7
›
Teams that deliver faster pay more attention to
operational analytic data. According to our survey,
nearly three-quarters (73%) of organizations delivering
monthly or more frequently are monitoring application
performance of servers or services, compared with only
63% of companies delivering quarterly or less frequently.
Those delivering faster are also more likely to track usage
and monitoring analytics. Overall, 54% of all respondents
reported using real-user monitoring. These methods
provide some insight into customer experiences, but
better insight into real customer experiences, tied back to
desired customer outcomes, is still needed.
Take action:

Use IaC to define development, testing, and
production environments.

Automate deployment processes to
improve speed and consistency.

Measure and manage technical debt.

Tie operational and analytics data back to
desired customer outcomes.
8
Smaller Releases Mean Faster
Releases
Smaller-scope releases enable organizations to release
more frequently. Faster releases let organizations try new
ideas quickly, gather feedback from customers, and then
adapt and improve (see Figure 8). Our survey found that:
›
›
Smaller releases are essential to faster application
delivery. Fifty-two percent of teams delivering monthly or
more often are in organizations with widespread use of
small, incremental delivery techniques, compared with
46% of teams delivering at a slower pace. Doing less in
each release reduces risk, increases speed of feedback,
and ultimately reduces waste because there are fewer
unknowns when it comes to what customers really want.
FIGURE 9
Fifty-Seven Percent Of DevOps Teams Delivering
Monthly Or More Prioritize Single-Piece Items To
Deliver Continuously
“How does your team plan and schedule work?”
Release less frequently (N = 256)
Release monthly or more frequently (N = 168)
In sprints
In single-piece product
backlog items, stackranked, delivered
continuously
38%
47%
49%
57%
By release and
milestone
Monthly or more frequent delivery requires
incremental, quick delivery. For those releasing this
quickly, 57% are releasing in single-piece, prioritized
product backlog items, delivered continuously. In addition,
47% are delivering in sprints (see Figure 9).
By release train
∆ 9%
59%
64%
36%
40%
∆ 8%
∆ 5%
∆ 4%
Base: 425 IT professionals involved in application development and
delivery
FIGURE 8
Software-Powered Value Delivery Cycle
Source: A commissioned study conducted by Forrester Consulting on
behalf of Microsoft, June 2015
Take action:
Customer
Experience
Systems of
record
Systems of
engagement
Solution
Co-creation
Source: Forrester Research, Inc.
Measured
Satisfaction
Gap
Systems of
automation

Reduce the scope of each release to
obtain faster feedback and reduce waste.

Reduce the scope of each release to
minimize risk by reducing dependencies
and simplifying deployments.

Reduce the scope of each release by
focusing each one on improving a single
outcome for a single persona.
9
Faster Feedback Means Better
Results
Regardless of delivery cadence, organizations want to
improve their delivery speed in order to improve customer or
user satisfaction. At the same time, organizations that
deliver faster are more confident that they will achieve
higher customer satisfaction. Our survey found that:
›
›
›
Adopting DevOps leads to greater customer
satisfaction and revenue. Sixty-nine percent of those
delivering at faster rates expect customer satisfaction to
improve. In addition, 56% of organizations delivering
rapidly expect rising revenue as delivery speeds increase,
compared with only 49% for slower deliverers (see Figure
10).
Adopting DevOps practices also leads to increased
efficiency. DevOps teams that release more frequently
expect to see improved efficiency between the IT
operations teams and development teams with faster,
more accurate delivery of applications (see Figure 10).
Adopting DevOps enables teams to better utilize
customer insights. Faster application delivery cycles
provide organizations with timely customer feedback. In
turn, the insight gained from frequent application delivery
enables teams to see how their work improves customer
experiences and ultimately satisfaction. These insights
drive new ideas that can be tested sooner, either
confirming customer experience improvements or
signaling that different approaches are needed.
Take action:

Measure customer satisfaction/business
outcomes tied to releases.

Measure and use customer feedback to
increase satisfaction and user experience.
FIGURE 10
Impact Of Faster Application Delivery
“What are the positive business impacts your
company would expect to get from faster,
more accurate delivery of applications?”
Release less frequently (N = 256)
Release monthly or more frequently (N = 168)
Increased revenue
49%
56%
Improved
efficiency between
operations and
development teams
59%
65%
63%
69%
Improved user
satisfaction
Distinctive
competitive
advantage
46%
43%
∆ 7%
∆ 6%
∆ 6%
∆ -3%
Base: 425 IT professionals involved in application development and
delivery
Source: A commissioned study conducted by Forrester Consulting on
behalf of Microsoft, June 2015
10
Key Recommendations
Faster application delivery requires embracing DevOps practices. Every organization can benefit from these practices;
however, each needs to assess its current situation and determine which practices are most immediately critical and
which can be planned for the future. Forrester’s in-depth surveys with application development and IT operation
professionals point to seven key recommendations for companies looking to adopt DevOps practices to improve their
results:
›
›
›
›
›
›
›
Streamline, simplify, and automate the delivery pipeline. Treating the delivery process as a pipeline process
and using practices like lean value stream analysis to understand the process, starting with the point where code
is checked in to the point where customers receive benefit, helps companies identify barriers to faster delivery. A
more complete view actually starts with the point where ideas or incidents are identified and flows through
customer experience.
Expand test automation to improve quality while increasing delivery speed. Manual testing cannot keep
pace with consumer demand for new capabilities. Leading organizations automate application testing using APIs
and incorporate this testing into their continuous integration processes. Removing dependence on manual testing
increases delivery speed, reduces manual errors, and reduces testing cost. Automating testing as part of the CI
process provides developers with fast feedback and ensures quality is built in from the beginning.
Use infrastructure as code and cloud technologies to simplify and streamline environment provisioning.
Developers need environments as soon as they start to build code, and testers need them soon after. Having the
right environments available when they are needed improves speed and quality by eliminating time spent waiting.
Standardizing environment configurations across development, testing, and production eliminates errors and
hard-to-duplicate incidents caused by undocumented changes across environments.
Reduce technical debt to increase responsiveness and reduce cost. Technical debt, or postponed remedial
technical work, accumulates over time and makes applications brittle, complex, and hard to change. Reducing
technical debt makes applications more modular, resilient, and capable of being released more frequently.
Decouple applications and architectures to simplify delivery activities. Modern applications are built around
APIs that decouple architectures and connect applications together. The resulting components and services can
be developed and delivered relatively independently, simplifying delivery activities, increasing delivery speed, and
reducing delivery risk.
Collect and analyze feedback to drive better requirements. Rapid delivery cycles enable organizations to
measure whether what they delivered mattered and to know what they should work on next. Rapid feedback
enables organizations to deliver increasing levels of customer delight while reducing the amount of time, money,
and effort they spend on building things that no one wants or needs.
Measure business outcomes tied to application releases. When rapid application delivery enables better
customer insight, organizations are able to see a close connection between business strategy, execution,
customer experience, and business results. Organizations try out new ideas, gather feedback, and rapidly refine
their solutions based on customer experiences. This enables organizations to identify and rapidly act upon new
business opportunities.
11
Appendix A: Methodology
In this study, Forrester conducted an online survey of 425 organizations in the US, the UK, France, Germany, China, India,
and Brazil to evaluate application development and delivery practices. Survey participants included decision-makers in
application development and IT operations. Questions provided to the participants explored current application development
practices, cycle times, and testing methods/environments. The study began in May 2015 and was completed in June 2015.
Appendix B: Supplemental Material
RELATED FORRESTER RESEARCH
“The Eight Tenets Of Faster Application Delivery,” Forrester Research, Inc., April 15, 2014
“Modern Application Delivery Demands A Modern Organization,” Forrester Research, Inc., June 27, 2014
“The Software-Powered Business,” Forrester Research, Inc., October 20, 2014
12
Appendix C: Demographics
FIGURE 11
Respondent Country And Company Size
Country
Company size
France
11%
United States
27%
Germany
11%
500 to 999 employees
4%
20,000 or more
employees
19%
India
11%
United
Kingdom
14%
Brazil
14%
China
12%
1,000 to 4,999
employees
53%
5,000 to 19,999
employees
24%
Base: 425 IT professionals involved in application development and delivery
(percentages may not total 100 because of rounding)
Source: A commissioned study conducted by Forrester Consulting on behalf of Microsoft, June 2015
FIGURE 12
Application Development And Delivery Involvement
“In your typical workday, how involved are you in each of the following tasks?”
This is my core job
function
I am involved with this task,
but it’s not my core job function
Software development strategic planning
Software development and testing
69%
59%
29%
39%
Software and infrastructure support and maintenance
54%
39%
Infrastructure provisioning and configuration
54%
39%
Software release and deployment
53%
41%
Base: 425 IT professionals involved in application development and delivery
Source: A commissioned study conducted by Forrester Consulting on behalf of Microsoft, June 2015