Application Security & DevOps
How DevOps can keep application
security from being an afterthought
A lot of Dev and Ops people look at their security teams with disdain. "Those
guys are just blockers... All they do is throw obstacles in my way and say no.
They'll never be DevOps or even Agile." In return many security people see the
emergence of DevOps as creating a wilder, less managed environment in which
they would face more risk and greater security challenges.
--- James Turnbull, Puppet Labs
I recognize that my code will be used in ways I cannot anticipate, in ways it was
not designed, and for longer than it was ever intended.
--- Rugged Manifesto (Josh Corman, David Rice, Jeff Williams)
My Observation: prevailing view of security as separate from
development and operations, DevOps or no DevOps.
Application Security is a 2nd Class Citizen
• PROBLEM EXAMPLES:
• SSL is a requirement, but may be addressed very
late in the SDLC. Coding for SSL and non-SSL may
differ substantially.
• LDAP authentication (authc) and/or authorization
(authz) is a requirement, but ops may not find
time to supply a dev/test directory server with
realistic entries. Or any dev identity server at all…
• Automated tests on web pages test happy-path
function, maybe even a few data validation edge
cases, but often ignore security tests.
Application Security is a 2nd Class
Citizen – cont’d
• No firewalls in dev – plenty of ‘em in prod. Arguments
over ports at 11th hour.
• Third-party libraries not subjected to security audits.
• Production system trust boundaries differ from
development
• DBAs and development team have widely diverging
views on authentication.
• Organizational security chief may have pet theories
about IT security – best ask early.
• End users want SSO…nobody asked them.
History of DevOps
• DevOps comes from a need for innovation on the systems side of
technology work.
• ESM (Enterprise Systems Management)
– mid-2000′s
– ITIL as a governance framework largely superseded by “ITIL Lite” Visible Ops
approach
– shift from “large vendor” focused to more open source and smaller vendor
offerings.
• first O’Reilly Velocity conference – 2008,
– focus on Web performance and operations
– 2009 seminal presentations about the developer/operations collaboration at
large. Provisioning tools like Puppet and Chef.
• Parallel thinking about “Agile Systems Administration
– Focus on analogies from Kanban/Lean manufacturing processes -> IT systems
administration.
– 2009: popularization of the term DevOps.
What is DevOps?
• More collaborative relationship between
development teams and operations teams
• System administrators participating in an agile
development process alongside developers
• Fusion of:
– Toolchains of monitoring and provisioning tools, CI,
automated testing etc
– agile processes
– Development teams /operations team cooperation
Secure SDLC
•
Security requirements need to be defined as early as possible during the SDLC
Standard Phase
Security Task
Notes
Kickoff
Organizational security
policy
Management support for
security
Requirements
Project security policy
Security model
Security Specification
Design
Defensive threat modeling
Development
Security code reviews
Includes TDD and unit test
Testing
Adversarial threat modeling
(pen-tests)
Functional, integration,
system, UAT
Implementation
Security configuration
Smoke tests
Ops & Maintenance
Security maintenance
checklist
Decommissioning
Back-out security
configuration
E.g. dedicated LDAP groups
Some Secure SDLC Examples
• Some processes and standards that can support security in a
SDLC:
–
–
–
–
Microsoft’s Trustworthy Computing Security Development Lifecycle
SEI Team Software Process for Secure Software Development (TSP)
Correctness by Construction
Merging security assurance activities into Agile development methods
(recent)
– The Common Criteria (ISO/IEC 15408)
– Software Assurance Maturity Model
• Many common processes and methodologies are not
designed specifically to address software security from the
ground up.
Problems with Secure SDLCs
•
•
•
•
One of the major hurdles is the availability of security expertise for the developer.
2009 Veracode survey:
– Only 13% of companies know the security quality of critical applications.
– 60% are increasing scrutiny on partners for delivering secure code
– Only 34% of companies have a comprehensive SDLC process which integrates
application security.
– 57% of organizations don’t have AppSec training programs for their
developers.
– 64% of respondents cannot afford desired application security.
SSE-CMM, Trusted CMM, Common Criteria, Correctness by Construction, and TSP
Secure require security-knowledgeable resources and that an organization adopt a
different and more rigorous development approach.
Organizations are showing increased response to security, but there is still a long
way to go before considerations of security in the SDLC can be considered
mainstream.
Typical Process Conflicts
•
Operations often not involved in architectural decisions or code reviews.
– E.g. development team comes up with some system interfaces that ops hears about well
after the fact. Whaddya mean you’ve got that external party coming in that way?
•
Developers often cannot identify minimal steps to reach a configuration state.
– E.g. trial and error to set up working security, little idea of what is actually needed.
– Developers not given time to determine essential elements of a vulnerability fix.
•
Developer configurations are optimized for development, not production.
– E.g. no LDAP, no SSL, inadequate development DBs.
– Development typically has no configuration management, test might have, prod usually
does.
– An entire developer system may be on one server, where prod uses four.
•
Development and operations do not cooperate to define the runtime environment
– E.g. prod web server may be entirely different than development servers.
•
Do not assume that Operations is familiar with the technology used by
development
Secure DevOps Challenges
• “Traditional” DevOps
– developers want change, operations wants
stability
– Both give security short shrift
– DevOps - mixing of infrastructure operations with
software development, in short release cycles –
squeezes out security, which requires deliberation
• Current DevOps is often a dyad
• Secure DevOps is a triad
Goals for Secure DevOps
• Agile Security Testing
• Secure Coding + Secure Operations + Secure
Collaboration
• Developer training now has meaning
• Faster communication between Development and
Operations on Vulnerability Information
• Faster patching/closure of vulnerabilities
• Process of collaboration between Development and
Operation
• Single manager/management system for security
during the release cycle
Goals for Secure DevOps (2)
• Identify security responsibilities:
– Operations is responsible for architecting the more stable
infrastructure components.
– Agile developers handle the application, which is now a 1st class
security concern along with infrastructure.
• Developers work within a secured environment that is locked down
to production standards.
• Establish a security standard for the (infrastructure) baseline that is
monitored from development to production.
• End results:
– accountability for securing the whole stack
– significantly reduce the impact of security audits on the DevOps cycle
– High degree of automation, hence efficiency
• Engage your QA/QC people
Automating Security
• What to automate
– SAST (Static Application Security Testing)
• Examine source code, byte code or application binaries for
conditions indicative of a security vulnerability
• Involves tools that help with either a) Review or b) Static
Analysis
– DAST (Dynamic Application Security Testing)
• Black-box (Functional and non-functional), White-box, and
Defect-based tests.
• Examine the application in its running state in order to
discover security vulnerabilities
• Robustness testing (i.e. fuzz testing) or fault-injection
Automating Security (2)
• Introduce security automation gradually
• Low-hanging fruit
– Add static code analysis to CI system, clearly
distinguish from non-security
• Require that functional tests incorporate authc
and authz
– E.g. QTP or Selenium web page tests must hook into a
security realm, even if it’s a mock.
– Must test method authorizations in development
• Engage your CI enthusiasts
Separation of Duties
• Deconflicting DevOps and traditional attitudes
– Management support for secure SDLC/secure
DevOps/Rugged DevOps process
– Single boss for security work across Dev and Ops,
with management and business backing
– IT Separation of Duties is fine as far as it goes, but
don’t let an overzealous admin misinterpret it.
Conclusions
• Identify and engage champions and process
enthusiasts
• Operations owns “static” infrastructure baseline;
development owns application
• Single manager overseeing security
• Developers have same environment as production
• Development, test and production all use security
• Build security into automated testing and provisioning
• Train your people – developers and operators – in
security
Notes
• DAST. Fault-injection can include Trusted
Computing Base (TCB) testing including the
loading of shared libraries or DLLs at run-time.
Dynamic testing can include concurrency
checking and many other factors.
• Tools:
– SAST: HP/Fortify, IBM, Veracode, Checkmarx,
Grammatech, Amorize, Coverity, Klocwork and
Parasoft, FindBugs, PMD and FxCop
– DAST: WebScarab
Resources
• http://architects.dzone.com/articles/short-history-devops
• https://buildsecurityin.uscert.gov/bsi/articles/knowledge/sdlc/326-BSI.html
• ITIL Lite, Malcolm Fry, http://www.best-managementpractice.com/Publications-Library/IT-Service-ManagementITIL/ITIL-2007-Edition/ITIL-Lite-A-Road-Map-toImplementing-Partial-or-Full-ITIL/
• http://www.infoq.com/news/2010/06/rugged-softwaremanifesto
• http://www.sans.edu/research/securitylaboratory/article/it-separation-duties