2019
State of
DevOps
Industry Report Card
Contents
Introduction
Financial services & insurance
Government
Retail
Technology
Telecom
Conclusion
1
5
11
17
24
30
37
Introduction
For the eighth annual State of DevOps Report, we surveyed
nearly 3,000 technical professionals around the world and across
industries. This report focused on one of the biggest challenges
organizations face as they expand their DevOps practice:
integrating security into the software delivery lifecycle.
While we found that integrating security is not always a
straightforward process, our research shows that the benefits
of doing so are real. Organizations with a high level of security
integration are not only more confident in their security posture
and view security as a shared responsibility across teams, but
are also able to deploy on demand more frequently, remediate
vulnerabilities faster, prioritize security improvements over feature
delivery, and halt a push to production to address a security issue.
But not all organizations are the same, and each industry has its
own unique challenges. Our objective is to provide pragmatic,
prescriptive guidance for organizations struggling to achieve
success with DevOps. That’s why we’re sharing our first-ever
industry report card.
1
I ntroduction
We examined the data by industry alongside what we know about
each industry's particular characteristics. For example:
• Are audits dominant in your work?
• Is the pressure to release consumer-facing features affecting
your security posture?
• Are your teams too isolated to be empowered to transform?
This revealed concrete measures for organizations in specific
industries to improve their DevOps practice by building security
in from the start.
Innovate with a security-first mindset
To determine the depth of integration of security into the
software delivery lifecycle, we asked survey respondents which of
these five phases security was involved in: requirements, design,
building, testing, and deployment. We categorized respondents’
answers into five levels of security integration:
•
•
•
•
•
2
Level 1: No integration of security in any of the phases
Level 2: Minimal integration (one of five phases)
Level 3: Selective integration (two of five phases)
Level 4: Significant integration (three or four of five phases)
Level 5: Full integration (all phases)
Industry report card
DEVOPS
SECURITY INTEGRATION
Retail
B
C+
Telecom
A-
B
Technology
A
A-
Government
C
B-
Financial services
& insurance
B
C-
Financial services
and insurance
With a desire for technology innovation across the industry
and a high degree of scrutiny on its security practices,
financial services and insurance firms should be expected
to implement forward-thinking security. Many of these
firms are stuck in the middle of their DevOps evolution,
with very few reaching the higher stages where teams are
successfully applying automation to security considerations.
REPORT CARD
DevOps
Security Integration
High achiever with
opportunity to grow
Need to better
apply themselves
B
C-
This is not surprising considering the significant challenges
firms in the financial services and insurance industry face.
Most of these companies have been around for decades
and are dealing with large amounts of technical debt,
complex organizational structures, and many functional
silos. Compliance and audits are expensive, time-consuming,
and disruptive to the business. Transforming a 100-year old
company looks very different from transforming a company
that has only been in existence for 10 years, and it requires
a massive shift in culture, processes, and practices.
3
Financial services and insurance
Challenges
Expanding DevOps practices
The financial services and insurance sector has the most
organizations that we categorize as Medium on the DevOps
evolutionary journey; conversely, it has the fewest organizations
we identify as High. We find that many organizations in this
industry have solid DevOps foundations to build upon, but face
challenges evolving to a high level.
Ability to deploy on demand
Across industries, organizations reap significant competitive
advantages the quicker they can deploy new features to
customers. The ability to deploy on demand to production is a
capability claimed by forty‑two percent of respondents within
financial services and insurance; more than those in telecom
(31 percent) or government (41 percent), but fewer than in retail
(57 percent) and tech firms (49 percent) by significant margins.
Fifty-nine percent of respondents agreed that technology
and processes limit their ability to deploy — representing
an opportunity for senior leadership to mandate more
standardization and agile methods of working.
4
DevOps evolution
HIGH LEVEL
MID LEVEL
LOW LEVEL
Telecom
17%
79%
Technology
16%
78%
6%
Retail
18%
75%
7%
Financial services
& insurance
8%
84%
Government
11%
75%
4%
8%
14%
Ability to deploy to production
ON DEMAND
DAILY UP TO 6X A WEEK
Most On-Demand
Industry (Retail)
Global Average
All Industries
Financial services
& insurance
Least On-Demand
Industry (Telecom)
WEEKLY OR LESS
16%
57%
46%
42%
31%
12%
17%
6%
20%
6%
22%
7%
Deploy rate limited by technology and process
RESPONSES FOR “STRONGLY AGREE” AND “AGREE”
Gov’t
Telecom
Fin Serv
& Ins.
Retail
Tech
64% 60% 59% 53% 53%
Financial services and insurance
Audit process
Audits can be disruptive events, particularly for
highly regulated industries that are subject to many
different kinds of audits. But are they actually helping
to decrease risk and improve security posture?
Only 17 percent of financial services and insurance
industry respondents strongly agreed with the
statement “Our audit process helps minimize risk to
the business.” That is the lowest of all industries.
Only 12 percent of respondents strongly agreed
that issues identified during the audit process were
prioritized by the business. Again, this is the lowest
of all industries with an overall average of 18 percent.
Audit practices
(RESPONSES FOR “STRONGLY AGREE”)
ALL INDUSTRIES
FINANCIAL SERVICES & INSURANCE
Our audit process helps minimize risk to the business
24%
17%
Issues identified in audit are prioritized by the business
18%
12%
Our audit and remediation process requires that
we stop normal work to get it done
14%
12%
5
Financial services and insurance
Opportunities
Integrate security earlier in the delivery cycle
Respondents from financial services and insurance
firms were about average when it came to integrating
security into all of the phases of the software delivery
lifecycle (requirements gathering, design, build, test,
and deployment). However, they were lowest in
integrating security into the requirements phase.
Involvement of security in the software delivery lifecycle
All
industries
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
When there’s an AD HOC REPORTED ISSUE IN PRODUCTION
57%
57%
59%
60%
57%
61%
44%
59%
35%
46%
54%
38%
47%
46%
36%
40%
36%
44%
46%
44%
40%
41%
When there’s a SCHEDULED AUDIT OF PRODUCTION
52%
53%
63%
54%
At the DEPLOYMENT PHASE of the delivery cycle
44%
47%
41%
At the TESTING PHASE of the delivery cycle
49%
54%
47%
At the BUILDING PHASE of the delivery cycle
41%
45%
41%
At the DESIGN PHASE of the delivery cycle
45%
50%
46%
At the REQUIREMENTS PHASE of the delivery cycle
40%
6
43%
38%
40%
Financial services and insurance
Relationship between Security and Dev/Ops
According to a study by IBM Science Institute,
the cost of fixing defects increases exponentially
as it progresses through the delivery lifecycle. Prevention
is key and collaborating with security teams when
identifying requirements can mitigate future risks and
significantly reduce development costs.
Sixty-seven percent of respondents in this industry
agreed with the statement “Our security team could
prevent unplanned work if included in the software
development life cycle earlier” — the highest positive
response of all industries. This shows that these teams
recognize the value in involving security earlier in
the cycle.
RESPONSES FOR “AGREE” AND “STRONGLY AGREE”
All
industries
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
Security is a shared responsibility across delivery and security teams
78%
81%
77%
77%
77%
72%
Security processes/policies significantly improve security posture
64%
70%
57%
59%
64%
57%
Security team could reduce unplanned work if included in software dev cycle earlier
62%
62%
67%
62%
63%
62%
Security is a major constraint on ability to deliver software quickly
41%
39%
38%
38%
48%
49%
Security team encounters a lot of friction when collaborating with delivery teams
39%
34%
43%
37%
51%
43%
Generally provide exceptions for security issues rather than addressing them
34%
30%
34%
34%
46%
31%
7
Financial services and insurance
Empower teams to build security in
The majority of financial services and insurance firms we surveyed have a
centralized security function. In a large enterprise, these security teams can
support hundreds of application development teams. Security is viewed as a
bottleneck and these teams are so buried in manual processes that they have
little time to invest in improvements.
Only 26 percent of financial services and insurance firms have designated
security experts embedded within the delivery teams. Having a security expert
on the team can ensure that security is treated as a design constraint and
eliminate bottlenecks to delivery.
Security function structure
All industries
We have a centralized security function
(e.g. InfoSec team) that supports
delivery teams on demand
We have a centralized security function,
and delivery teams also include a designated
security expert (e.g., a security champion
that resides in the scrum team)
We have a decentralized security function
(e.g. each delivery team has their own
security expert that reports to that team)
8
Technology
Financial
services
& insurance
Retail
Telecom
Government
48% 44% 61% 54% 41% 54%
31%
32% 26% 28% 41% 30%
14%
19% 9% 11% 14% 9%
Financial services and insurance
Prioritize security improvements
Only 33 percent of financial services and insurance
firms are able to prioritize automating security
controls over feature delivery — the second lowest
of all industries. According to the CIS Top 20 Critical
Controls, automating security configurations for
hardware and software is one of the most effective
and foundational practices to prevent attacks.
Only 21 percent of these firms are able to prioritize
compliance reporting and proof over feature delivery
— the lowest of all industries.
Firms that are serious about improving their security
posture should be investing in more automation, and
senior leadership needs to make this a top priority.
Automation will make it easier to harden infrastructure
and application configurations and prove compliance,
which in turn will reduce audit time and give teams
time back to work on higher impact initiatives.
VOICE OF THE INDUSTRY
“We are providing a lot of DevOps functionality. We’ve gone from
taking 24 to 48 hours to set up a development environment to
where a typical server build now takes 20 minutes. There’s a
setup for each type of server — A, B, or C; whichever flavor the
development team needs, they know we will be able to deliver.
Every 15 minutes they get back in their day, they can roll into
more development work.”
Pope Davis, senior director of systems engineering, NYSE/ICE
9
Government
Government is a broad category that includes municipal
departments and national government agencies.
Our survey spread the net across the globe and across
levels of government. Fifty‑four percent of respondents
were in the US or Canada, and 28 percent were in
Europe. It is likely that the cutting-edge efforts of
government agencies like the United States Air Force
are on one side of the scale while understaffed and
isolated local governments are on the other.
REPORT CARD
DevOps
Security Integration
Should do better
Strong performance
(surprisingly)
C
10
B-
G
 overnment
Each sub-category of the government industry has its
own unique challenges, of course, but all are striving
to modernize their applications and infrastructure,
reduce costly project overruns, ensure security and
compliance, all in the face of shrinking budgets.
From one perspective, the government sector leads
the way in security integration. Forty-three percent
report either significant integration or full integration.
From the very same data set, government is nearly at
the bottom, with 42 percent having no integration or
minimal integration. There is not much middle ground
for the government, it seems.
Security integration model
LEVEL 1
No integration
LEVEL 2
Minimal
integration
LEVEL 3
Selective
integration
LEVEL 4
Significant
integration
LEVEL 5
Full integration
28%
26%
16%
Government
16%
Financial services & insurance
15%
Retail
Technology
Telecom
11
G
 overnment
Challenges
Deployment frequency
Only 41 percent of government agencies
can deploy on demand, coming in second to
last for all industries. Government also had
the highest number of respondents report
that they were able to deploy to production
between once a week and three times a month
(15 percent) and once a month (14 percent).
Deployment frequency
ON DEMAND
2X PER DAY
OR MORE
1X PER DAY TO
6X PER WEEK
1X PER WEEK TO
3X PER MONTH
1X PER MONTH
OR LESS
FREQUENCY AT WHICH WE CAN DEPLOY
46%
All industries
49%
Technology
Financial services
& insurance
20%
15%
31%
27%
41%
12% 6%
16% 12%
22%
11%
12% 6%
15% 12% 6%
20%
57%
Government
17%
18%
42%
Retail
Telecom
19%
20%
13% 7%
15% 13%
FREQUENCY AT WHICH WE ACTUALLY DEPLOY
All industries
24%
Technology
26%
Financial services
& insurance
Retail
Telecom
Government
12
17%
14%
14%
13%
28%
16%
17%
19%
19%
13%
17%
20% 7% 15%
12%
6%
25%
16%
33%
20%
25%
18%
29%
25%
29%
10%
17%
29%
G
 overnment
Time to remediate vulnerabilities
Government agencies also reported the slowest time to
remediate critical vulnerabilities with only three percent
of respondents being able to remediate in an hour or
less and 24 percent able to remediate in less than
one day.
Typical time to remediate critical security vulnerabilities
All
industries
Tech
FinServ &
insurance
Retail
Telecom
7%
34%
12%
6%
3%
30%
41%
28%
24%
When there’s an AD HOC REPORTED ISSUE IN PRODUCTION
32%
27%
31%
31%
When there’s a SCHEDULED AUDIT OF PRODUCTION
14%
10%
17%
13%
At the DEPLOYMENT PHASE of the delivery cycle
11%
4%
7%
17%
4%
4%
5%
4%
1 day to less than 1 week
33%
34%
1 week to less than 2 weeks
13%
12%
2 weeks to less than 1 month
8%
8%
1 month to less than 3 months
4%
4%
3 months to less than 6 months
2%
1%
2%
2%
2%
4%
All
industries
57%
52%
44%
Tech
57%
53%
47%
FinServ &
insurance
Retail
59%
60%
63%
Gov’t
57%
61%
44%
59%
35%
46%
54%
38%
47%
46%
36%
40%
36%
44%
46%
44%
40%
41%
54%
41%
Telecom
At the TESTING PHASE of the delivery cycle
49%
54%
47%
At the BUILDING PHASE of the delivery cycle
41%
45%
41%
At the DESIGN PHASE of the delivery cycle
45%
50%
46%
At the REQUIREMENTS PHASE of the delivery cycle
6 months or more
1%
Involvement of security in the software delivery lifecycle
6%
1 hour to less than 1 day
32%
Government agencies were below average at
integrating security into the early phases of the
software delivery lifecycle — requirements gathering,
design, build, and test — including having the
lowest percentage of firms with security integrated
into the build and design phases.
Gov’t
Less than 1 hour
7%
Security integration in the
early phases of the delivery cycle
1%
2%
0%
4%
4%
40%
43%
38%
40%
13
G
 overnment
Opportunities
Integrate security earlier in the
software delivery cycle
Out of all industries, government had the lowest
percentage of respondents reporting that security
tools are integrated into the development ecosystem,
and security and development teams collaborate on
threat models (9 percent).
stakeholders explore potential threats and brainstorm
countermeasures to mitigate them. This happens in
the earliest stage of planning and design, informs the
security testing plan, and builds trust between all of
the teams involved.
According to our 2019 State of DevOps report,
these two practices had the greatest overall impact
on improving confidence in security posture.
Government agencies were also below average across
the board when it comes to other practices that have
a high impact on security posture, including security
requirements being prioritized as part of the product
backlog (12 percent) and security experts evaluating
automated tests (13 percent).
Integrating security tools into the development
pipeline empowers developers to find and fix
security issues so they don’t inadvertently end up in
production. Though it may sound expensive to adopt
new tooling and practices, as noted earlier, fixing
defects earlier in the delivery lifecycle is much cheaper
in the long run, reducing development time and costs.
As these responses show, the biggest opportunity for
government agencies is integrating security earlier
in the delivery cycle. Threat models, for example,
are a collaborative exercise where representatives
from delivery teams, security, and business
14
G
 overnment
VOICE OF THE INDUSTRY
“We have a heavy focus on security and compliance because
we have many clients in the public sector. Automation is a
key component for us to be federally compliant. It allows us
to ensure our clients obtain and keep their Authority to
Operate (ATO).”
Bryan Belanger, principal consultant, Fervid
15
Retail
Retail and ecommerce businesses have seen
a lot of disruption over the past few years with
new business models quickly overtaking more
traditional ones. Many brand name retailers have
also made the news for massive data breaches
affecting billions of customers and costing these
businesses billions of dollars in fines.
REPORT CARD
DevOps
Security Integration
High Achiever;
opportunity to grow
Needs improvement
B
16
C+
R
 etail
An encouraging finding for retail is the increase from
2018 to 2019 in the proportion of firms we characterize
as highly evolved; tying with telecom, retail had the
greatest gain with nine points. Retail leads the pack
in 2019 with 18 percent of firms in the High category.
DevOps evolution level
HIGH LEVEL
MID LEVEL
LOW LEVEL
2019 RESPONSES
2018 RESPONSES
Retail
18%
75%
Telecom
17%
79%
Technology
17%
78%
Government
11%
75%
Financial services
& insurance
8%
84%
7%
5%
6%
14%
8%
Retail
9%
82%
9%
Telecom
8%
79%
13%
Technology
12%
Government
7%
Financial services
& insurance
9%
80%
78%
80%
8%
14%
11%
17
R
 etail
Deployment frequency
ON DEMAND
2X PER DAY
OR MORE
1X PER DAY TO
6X PER WEEK
1X PER WEEK TO
3X PER MONTH
1X PER MONTH
OR LESS
FREQUENCY AT WHICH WE CAN DEPLOY
46%
All industries
49%
Technology
Financial services
& insurance
20%
12%
6%
15% 12%
6%
12%
5%
20%
57%
15%
31%
27%
41%
Government
17%
18%
42%
Retail
Telecom
19%
16% 12%
22%
11%
20%
13%
7%
In 2019, retail found another bright spot: it has
the highest percentage of firms that can and do
deploy on demand — 57 percent are capable of
deploying to production on demand and 28 percent
say that they actually do deploy on demand.
That is head and shoulders above other industries.
The retail industry resolves their critical vulnerabilities
the fastest, with 53 percent reporting remediation in
under one day.
15% 14%
FREQUENCY AT WHICH WE ACTUALLY DEPLOY
All industries
24%
Technology
26%
Financial services
& insurance
Retail
Telecom
Government
18
17%
14%
14%
13%
28%
16%
20%
19%
19%
14%
17%
7%
19%
17%
25%
16%
33%
20%
25%
15%
26%
18%
29%
26%
29%
10%
17%
29%
Typical time to remediate critical security vulnerabilities
All
industries
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
7%
6%
12%
6%
3%
30%
41%
28%
24%
Less than 1 hour
7%
1 hour to less than 1 day
32%
34%
R
 etail
Challenges
Deeper security integration
Despite having the highest proportion of highly evolved
DevOps firms, retail lags behind other industries
when it comes to deeper levels of security integration.
Respondents that reported “significant” or “full”
security integration is just 32 percent, the lowest of all
industries. It also has the largest incidence of firms that
have no integration with security.
Retail is much less likely to integrate security into the
later stages of the delivery lifecycle (building, testing,
and deployment stages) compared to other industries.
While there is no single path for integrating security
into the software delivery lifecycle, achieving higher
levels of security integration enables teams to move
from a reactive to proactive security stance.
Security integration model
Involvement of security in the software delivery lifecycle
LEVEL 1
No integration
LEVEL 2
Minimal
integration
LEVEL 3
Selective
integration
LEVEL 4
Significant
integration
LEVEL 5
Full integration
All
industries
Tech
FinServ &
Insurance
Retail
When there’s an AD HOC REPORTED ISSUE IN PRODUCTION
57%
57%
59%
60%
52%
53%
63%
61%
44%
59%
35%
46%
54%
38%
47%
46%
36%
40%
36%
44%
46%
44%
44%
47%
41%
At the TESTING PHASE of the delivery cycle
20%
49%
20%
54%
47%
At the BUILDING PHASE of the delivery cycle
41%
45%
41%
At the DESIGN PHASE of the delivery cycle
12%
Government
40%
41%
54%
At the DEPLOYMENT PHASE of the delivery cycle
24%
Financial services & insurance
Retail
Technology
Telecom
45%
50%
46%
At the REQUIREMENTS PHASE of the delivery cycle
40%
43%
38%
Gov’t
57%
When there’s a SCHEDULED AUDIT OF PRODUCTION
24%
Telecom
40%
19
R
 etail
Opportunities
Improving security testing practices
In the traditional approach to application security,
code is pushed to a user acceptance testing (UAT) or
staging environment where the security team manually
inspects it. Inevitably, they discover an issue that
requires a fix, kicking off the whole cycle again. Move
testing to earlier in the software delivery lifecycle, and
bugs are found earlier when they are much cheaper to
fix. You can automate routine security checks, as you
would automate routine quality tests
Retail firms were below average when it came to
security testing practices, including static code
analysis, dependency checking, penetration testing,
and having security experts evaluate automated tests.
Automation of routine security checks gives time back
to security teams to review high-risk areas of the code
(such as authentication systems, cryptography, etc.),
evaluate tests when the application or environment
changes, and look for edge cases and collaborate with
delivery teams to secure and test applications during
development and deployment.
20
Vulnerabilities can live anywhere in the stack: in the
application code, third party components, APIs, servers,
databases, network devices, firewalls, cloud storage,
etc. The potential attack surface is vast, making
automated testing for known weaknesses your best
defense against attacks.
Frequency of security process & practices during testing
All
industries
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
24%
19%
26%
23%
Static code analysis
24%
29%
Dependency checkers (e.g., tools that check for the latest version
of npm packages, RubyGems, etc)
18%
22%
16%
16%
19%
14%
Penetration testing (e.g., vulnerability trigger or hacker tool testing)
17%
20%
11%
16%
18%
20%
Domain specific tests (e.g., tests that assess application with security-aware context,
such as the way your application does authentication or has access to data)
17%
20%
13%
21%
21%
16%
13%
17%
13%
Security experts evaluate automated tests
14%
17%
7%
R
 etail
Closing the gap between ability to
deploy and actual deployments
Deployment frequency
Retail firms have a strong ability to deploy, but the
29-point gap between capability and actual frequency
of deployment is the widest of all industries. Retailers
that want to get ahead of their competition need to be
able to deliver value to customers faster, but in a secure
and reliable way.
Fifty-three percent of respondents agreed that their
deployment frequency was limited by their technology
and processes. Standardizing on the technology stack
and modernizing processes is the way forward for
retail firms.
Deploy freq. limited by technology and process
RESPONSES FOR “STRONGLY AGREE” AND “AGREE”
Gov’t
Telecom
FinServ
& insurance
Retail
Tech
64% 60% 59% 53% 53%
ON DEMAND
2X PER DAY
OR MORE
1X PER DAY TO
6X PER WEEK
1X PER WEEK TO
3X PER MONTH
1X PER MONTH
OR LESS
FREQUENCY AT WHICH WE CAN DEPLOY
46%
All industries
49%
Technology
Financial services
& insurance
20%
12%
6%
15% 12%
6%
12%
5%
20%
57%
15%
31%
27%
41%
Government
17%
18%
42%
Retail
Telecom
19%
16% 12%
22%
11%
20%
13%
7%
15% 14%
FREQUENCY AT WHICH WE ACTUALLY DEPLOY
All industries
24%
Technology
26%
Financial services
& insurance
Retail
Telecom
Government
17%
14%
14%
13%
28%
16%
20%
19%
19%
14%
17%
7%
19%
17%
25%
16%
33%
20%
25%
15%
26%
18%
29%
26%
29%
10%
17%
29%
21
R
 etail
Increasing usage of high impact
security practices
Security practice adoption:
retail vs. all industries
(RESPONSES FOR “ALWAYS” OR “OFTEN” PERFORM PRACTICE)
ABOVE AVG PERFORMANCE
LOWER IMPORTANCE
Developers can provision
security hardened
infrastructure stack
on demand
ABOVE AVG PERFORMANCE
HIGHER IMPORTANCE
• Infrastructure-related security policies are tested and
reviewed before deployment.
• Security requirements are prioritized as part of the
product backlog.
Infrastructure
provisioned/configured
automatically using
security-approved
procedures
Domain-specific tests
Penetration
testing
ALL INDSTRIES AVERAGE
Dependency
checkers
Security personnel
Security tools integrated
Security review occurs
review/approve
into the dev ecosystem so
after new application
major code changes
developers can implement
code released to
before deployment
security features during
production
development phase
Static
Security and dev
code analysis
Security personnel
Infrastructure-related security teams collaborate
review/approve
policies tested/reviewed
on threat models
minor code changes
before deployment
before deployment
Security requirements
prioritized as part of
product backlog
Security requirements
tested as design constraint
Security experts evaluate
automated tests
LESS THAN AVG PERFORMANCE
LOWER IMPORTANCE
22
In our 2019 State of DevOps Report, we examined
the practices that had an impact on an organization’s
confidence in its security posture and how frequently
they are used. The practices that are both higher in use
and have a higher impact on security posture are:
LESS THAN AVG PERFORMANCE
HIGHER IMPORTANCE
The practices that are used less frequently and have a
high impact on security posture are:
• Security personnel review / approve major code
changes before deployment.
• Security experts evaluate automated tests.
• Security tools integrate into the development
ecosystem so developers can implement security
features during the development phase.
• Security and dev teams collaborate on threat models.
Retail respondents reported below-average usage of all
of these high-impact practices. Increasing usage of these
practices should be a key focus for retailers looking to
improve collaboration across teams and make security
easier to implement, more agile, and more iterative.
R
 etail
VOICE OF THE INDUSTRY
“Everyone wants to have stability while still being
fast; that’s one of the main objectives we have for
configuration management. Now we can quickly
build and configure systems that are standardized
— and when you become standardized, you become
more stable.”
Tom Sabin, manager for cloud and automation, Staples
23
Technology
Since the early days of DevOps when Flickr’s 10 deploys
per day shook the industry, tech companies have been
leading the charge on DevOps practices. So it’s no
surprise that tech companies have the highest grade of all
industries. Technology had the lowest percentage of firms
with no security integration (13 percent) and the highest
percentage of firms with full integration (17 percent).
Security integration model
LEVEL 1
No integration
LEVEL 2
Minimal
integration
Selective
integration
LEVEL 4
Significant
integration
DevOps
Security Integration
Teacher's pet
Exceeds
expectations
LEVEL 5
Full integration
25%
24%
REPORT CARD
A
LEVEL 3
21%
A-
17%
13%
Government
24
Financial services & insurance
Retail
Technology
Telecom
T
 echnology
You’re probably wondering: What sets tech companies
apart? Sixty-six percent of technology respondents
say that leadership supports DevOps initiatives always
or most of the time, the highest of any industry.
Though many DevOps initiatives start as grassroots
movements within an organization, at some point
leadership support is necessary to help it thrive.
Tech companies are also leading the pack in
integrating security into every phase of the
software delivery lifecycle. Half of all respondents
from technology firms are integrating security at
the design phase. In fact, technology firms lead
the way in integrating security for requirements,
design, building, and testing.
Support from senior leadership for DevOps initiatives
RESPONSES FOR “ALWAYS” AND “OFTEN”
All industries
Tech
59%
66%
Involvement of security in the software delivery lifecycle
All
industries
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
When there’s an AD HOC REPORTED ISSUE IN PRODUCTION
57% 57% 59% 60% 57%
61%
When there’s a SCHEDULED AUDIT OF PRODUCTION
52% 53% 63% 54% 44% 59%
At the DEPLOYMENT PHASE of the delivery cycle
44% 47%
41%
35% 46% 54%
At the TESTING PHASE of the delivery cycle
49% 54% 47% 38% 47% 46%
At the BUILDING PHASE of the delivery cycle
41%
45%
41%
36% 40% 36%
At the DESIGN PHASE of the delivery cycle
45% 50% 46% 44% 46% 44%
At the REQUIREMENTS PHASE of the delivery cycle
40% 43% 38% 40% 40% 41%
25
T
 echnology
All teams take responsibility for security at tech firms.
This sector was above the global average and led all
industries for four of the five categories (application
development, DevOps teams, quality teams, and
security teams). Taking on that responsibility builds
confidence in the overall security posture and reflects
cross-team collaboration.
As a result of the high level of security integration
amongst technology firms, respondents also report
that their security practices and policies significantly
improve their security posture and that security is
viewed as a shared responsibility across teams. They
experience the least friction when security teams
collaborate with delivery teams and provide the least
exceptions for security issues.
Level of responsibility for security
Relationship between security and Dev/Ops
RESPONSES: “FULL RESPONSIBILITY” AND “MOST RESPONSIBILITY”
RESPONSES FOR “STRONGLY AGREE”
All
industries
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
Application Development Teams
56%
59%
56%
53%
56%
50%
Infrastructure Development Teams
61%
63%
57%
62%
66%
59%
DevOps Teams
55%
60%
47%
56%
57%
44%
Quality Teams
41%
44%
35%
38%
43%
28%
Security Teams
73%
73%
74%
71%
70%
70%
All
Industries
Tech
FinServ &
Insurance
Retail
Telecom
Gov’t
Security is a shared responsibility across delivery and security teams
32%
35%
31%
28%
33%
32%
Security processes/policies significantly improve security posture
20%
23%
14%
22%
22%
13%
Security team could reduce unplanned work if included in software dev cycle earlier
20%
21%
21%
19%
21%
16%
Security is a major constraint on ability to deliver software quickly
13%
13%
14%
12%
20%
16%
Security team encounters a lot of friction when collaborating with delivery teams
11%
8%
12%
12%
19%
11%
Generally provide exceptions for security issues rather than addressing them
9%
26
8%
7%
10%
17%
9%
T
 echnology
Typical time to remediate critical security vulnerabilities
All
industries
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
Challenges
7%
6%
12%
6%
3%
Remediating vulnerabilities
30%
41%
28%
24%
32%
27%
31%
31%
14%
10%
17%
13%
11%
4%
7%
17%
4%
4%
5%
4%
1%
2%
2%
2%
4%
1%
2%
0%
4%
4%
Less than 1 hour
7%
1 hour to less than 1 day
32%
34%
1 day to less than 1 week
33%
34%
1 week to less than 2 weeks
13%
12%
2 weeks to less than 1 month
8%
8%
1 month to less than 3 months
4%
4%
3 months to less than 6 months
2%
6 months or more
1%
Considering how well-integrated security is, we had
expected technology firms to have significantly
higher than average remediation times. That was
not the case. Respondents from technology firms
reported average remediation times for critical
vulnerabilities. Seven percent are able to remediate
a critical vulnerability in less than one hour,
34 percent in less than one day, and 34 percent
in less than one week. These results are not bad,
but they could certainly be better. This indicates
bottlenecks in the vulnerability remediation process
that technology firms would be wise to address.
27
T
 echnology
Opportunities
Innovate with a security-first mindset
It probably comes as no surprise that technology companies
lead the way in usage of modern tools and practices. They
have adopted containerization and continuous delivery at a
very high rate. Almost half of those surveyed are working on
at least one containerized application. Nineteen percent are
always using continuous delivery.
Given technology firms’ early adoption of cutting-edge
tools and practices, we would expect them to also be above
average in container security. Yet they lag behind retail and
telecom for most practices, though, including:
• Container images automatically tested as part of the
continuous integration process service you work on
• Container images scanned for vulnerabilities
• Container images deployed if they adhere to security
and compliance policies
With new platforms, more applications, new employees,
and new services come new risks and attack surfaces.
Ongoing security practices need to be durable as
processes and technologies change.
28
VOICE OF THE INDUSTRY
“We’re not mired down in toil work doing rote,
repetitive, error-prone tasks. We’re focused on both
the high-value problems we solve at Splunk and
those problems we solve for our customer.”
Chris Vervais, director, site reliability engineering
Splunk
Telecom
Telecom development shops are under enormous
pressure to ship new features faster. Focusing on
security integration in these conditions often means
making trade-offs between improving security practices
and delivering new features to customers. As our data
shows, the temporary trade-offs these companies make
to improve their security practices will help them realize
faster and more frequent deployments that are also
higher quality and more secure and reliable.
REPORT CARD
DevOps
Security Integration
Exceeds
expectations
Hard worker
A-
B
29
T
 elecom
DevOps evolution level
HIGH LEVEL
MID LEVEL
LOW LEVEL
2019 RESPONSES
Retail
18%
75%
7%
Telecom
17%
79%
5%
Technology
17%
78%
6%
Government
11%
75%
14%
Financial services
& insurance
8%
84%
8%
Retail
9%
82%
9%
Telecom
8%
79%
13%
Technology
12%
Government
7%
Financial services
& insurance
9%
2018 RESPONSES
30
80%
78%
80%
8%
14%
11%
The telecom industry has seen big strides in
DevOps evolution in just one year. Like retail and
ecommerce, telecom also saw a big increase in the
proportion of firms we characterize as High on the
DevOps evolutionary journey. From 2018 to 2019,
the proportion of highly evolved telecom firms
increased by nine points from eight to 17 percent.
The number of organizations that scored in the Low
category dropped from 13 percent to five percent.
T
 elecom
A notably large proportion of telecom organizations are
embedding a designated security expert in teams while
maintaining a centralized security function (41 percent).
This can assure that security is treated as a design
requirement and is considered throughout the software
delivery cycle.
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
We have a centralized security function (e.g. InfoSec team)
who support delivery teams on demand
48% 44% 61%
54%
32% 26% 28%
41%
54%
41% 30%
We have a decentralized security function (e.g. each delivery team
has their own security expert that reports to that team)
14%
19%
9%
11%
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
When there’s an AD HOC REPORTED ISSUE IN PRODUCTION
57% 57% 59% 60% 57%
61%
When there’s a SCHEDULED AUDIT OF PRODUCTION
We have a centralized security function and delivery teams also include a designated
security expert (e.g., a security champion that resides in the scrum team)
31%
Involvement of security in the software delivery lifecycle
All
industries
Security function structure
All
industries
We should note that they score just about average
at involving security in the different aspects of the
software delivery cycle, though. Perhaps that is a
lagging indicator to watch in the future.
14%
9%
52% 53% 63% 54% 44% 59%
At the DEPLOYMENT PHASE of the delivery cycle
44% 47%
41%
35% 46% 54%
At the TESTING PHASE of the delivery cycle
49% 54% 47% 38% 47% 46%
At the BUILDING PHASE of the delivery cycle
41%
45%
41%
36% 40% 36%
At the DESIGN PHASE of the delivery cycle
45% 50% 46% 44% 46% 44%
At the REQUIREMENTS PHASE of the delivery cycle
40% 43% 38% 40% 40% 41%
31
T
 elecom
Challenges
Moving beyond the messy
middle stages of integration
While security integration does lead to great outcomes,
the path to them isn’t necessarily easy. In the 2019
State of DevOps Report, we’ve noticed a pattern in
Levels 2 and 3 where things get worse before they
get better. This is known as the J-curve, and it’s
what makes muddling through the middle level
so challenging.
With the majority of telecom respondents in Level 3
of security integration, it’s important to keep in mind
that this is a natural and temporary transition phase
during the evolutionary process. Increased friction
between teams and slower-than-expected delivery
is common at this level.
32
Security integration model
LEVEL 1
No integration
LEVEL 2
Minimal
integration
LEVEL 3
Selective
integration
LEVEL 4
Significant
integration
28%
LEVEL 5
Full integration
27%
19%
16%
11%
Government
Financial Services & Insurance
Retail
Technology
Telecom
T
 elecom
Relationship between Security and Dev/Ops
RESPONSES FOR “STRONGLY AGREE”
All
industries
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
Security is a shared responsibility across delivery and security teams
32%
35%
31%
28%
33%
32%
High-friction environment:
business and security
13%
At a far higher rate than other organizations,
19 percent of telecom firms report friction when
collaborating with security teams. They also
report that security is a constraint on software
delivery at a much higher rate (20 percent vs
13 percent average). Again, this points to the
fact that so many of our telecom respondents
are in Level 3 where friction between security
and delivery teams is the highest overall.
Security processes/policies significantly improve security posture
20%
23%
14%
22%
22%
Security team could reduce unplanned work if included in software dev cycle earlier
20%
21%
21%
19%
21%
16%
Security is a major constraint on ability to deliver software quickly
13%
13%
14%
12%
20%
16%
Security team encounters a lot of friction when collaborating with delivery teams
11%
8%
12%
12%
19%
11%
Generally provide exceptions for security issues rather than addressing them
9%
8%
7%
10%
17%
9%
33
T
 elecom
Deployment frequency
ON DEMAND
Ability to deploy on demand
2X PER DAY
OR MORE
1X PER DAY TO
6X PER WEEK
1X PER WEEK TO
3X PER MONTH
1X PER MONTH
OR LESS
FREQUENCY AT WHICH WE CAN DEPLOY
46%
All industries
49%
Technology
Financial services
& insurance
20%
15%
31%
27%
41%
12% 6%
16% 12%
22%
11%
12% 6%
15% 12% 6%
20%
57%
Government
17%
18%
42%
Retail
Telecom
19%
20%
13% 7%
15% 13%
FREQUENCY AT WHICH WE ACTUALLY DEPLOY
All industries
24%
Technology
26%
Financial services
& insurance
Retail
Telecom
Government
17%
14%
14%
13%
28%
16%
17%
19%
19%
13%
17%
20% 7% 15%
12%
6%
25%
16%
33%
20%
25%
18%
29%
25%
29%
10%
17%
29%
Being able to deploy on demand is a critical measure
of an organization’s agility. Of all industries, telecom
had the lowest ability to deploy to production on
demand (31 percent) and also the lowest frequency of
actually deploying on demand (16 percent).
Again, we suspect this is a result of many telecom
firms being at Level 3 of security integration. In
our 2019 State of DevOps Report, we found that at
Level 3, on‑demand deployment capability declines
significantly before it improves again in Levels 4 and 5.
A large percentage of telecom respondents strongly
agree that deployment frequency is limited both by
business needs (64 percent) and their technology and
process (60 percent).
Deploy freq. limited by business needs
RESPONSES FOR “STRONGLY AGREE” AND “AGREE”
Gov’t
Telecom
FinServ
& insurance
Retail
Tech
52%
64%
51%
50%
57%
Deploy freq. limited by technology and process
RESPONSES FOR “STRONGLY AGREE” AND “AGREE”
34
Gov’t
Telecom
FinServ
& insurance
Retail
Tech
64%
60%
59%
53%
53%
T
 elecom
Disruptive, unplanned work
The tension between shipping new features and
improving security is felt most for telecom firms when
they discover a critical or high-severity issue right
before deploying to production. Telecom firms were
least likely to halt a push to production after finding a
critical or high severity security issue.
Telecom respondents were also much more likely to
strongly agree that audit and remediation processes
require that they stop normal work to get it done.
Level of security vulnerability that would stop production
FinServ &
All
Tech
insurance
industries
Critical severity security issues
Retail
Telecom
Gov’t
82% 83% 88% 83% 79% 84%
High severity security issues
75% 78% 75%
72% 66% 75%
Medium severity security issues
35% 37% 33% 34% 35% 33%
8%
RESPONSES FOR “STRONGLY AGREE”
All
industries
Tech
FinServ &
insurance
Retail
Telecom
Gov’t
Our audit process helps minimize risk to the business
24% 27%
17%
22% 23% 22%
Issues identified in audit are prioritized by the business
18%
18%
12%
24% 24%
15%
Our audit and remediation process requires that we stop normal work to get it done
Low severity security issues
6%
Sentiment after audit
7%
6%
5%
3%
14%
14%
12%
14%
23%
9%
35
T
 elecom
Opportunities
36
Pushing beyond the messy middle
Reducing unplanned work
For telecom firms that are in Stages 2 or 3, we
encourage them to stick through it because there
is a light at the end of the tunnel. The process of
identification, creation, growth and refinement has to be
repeated many times across each of these stages, and
it involves a myriad of parties, depending on the nature
of each process — change control boards, monitoring
experts, policy departments, compliance reviews,
auditors, user research, production control and more.
Each process requires a feedback cycle, and there are
several processes. Evolution will naturally take some
time, but the outcomes of this effort will help propel
telecoms to the higher stages of evolution.
Unplanned work includes any break/fix work,
emergency software deployments and patches,
responding to urgent audit documentation requests,
and so forth. In our 2016 State of DevOps Report
we found that high-performing organizations spend
22 percent less time on unplanned work and rework.
As a result, they are able to spend 29 percent more
time on new work, such as new features or code.
Furthermore, continuous delivery predicts lower
levels of unplanned work and rework in a statistically
significant way.
They are able to do this because they build quality
and security into each stage of the delivery cycle
through the use of continuous delivery practices,
instead of retrofitting quality and security at the end
of a development cycle. Disruptive, unplanned work is
holding telecom firms back and they should invest in
identifying causes of unplanned work and automate
more of the delivery pipeline to ensure that they can
deploy fixes quickly.
T
 elecom
VOICE OF THE INDUSTRY
“If you do tasks manually, there will always be mistakes
somewhere. If you automate, you eliminate the mistakes
of human error.”
Roman Frei, systems engineer, Swisscom
37
Conclusion
DevOps, as we so frequently say, is a journey, not a destination.
By providing an industry-specific view into the state of DevOps
practices today, we hope to help more organizations design
effective strategies based on where they are today and their
unique challenges.
Successful DevOps practices help the different teams that are
involved with software delivery feel more like colleagues and less
like adversaries. It’s the same with integrating security: it fosters a
feeling of cooperation among the security, dev, and ops teams. The
highly evolved teams we encountered in the 2019 State of DevOps
survey were not simply shifting security left. They had cultivated
a powerful blend of high-trust environments, autonomous teams,
and a high degree of automation and cross-functional collaboration
among application teams, operations, and security teams.
These outcomes are achievable by any organization in any
industry and we hope this report has provided a roadmap to help
you integrate security more deeply into your DevOps practice.
To learn more, read the 2019 State of DevOps Report.
38