Why do developers make dangerous software errors?
Michele Moss and Nadya Bartol
OWASP AppSec USA
September 23, 2011
Key questions
Who asks developers to develop secure code?
Do developers know why they need to develop secure code?
What should developers do/not do to develop secure code?
Can developers develop secure code by themselves?
How do developers know they have developed secure code?
Why should developers care?
Next Steps?
1
Requirements for secure code are implicitly and not explicitly
stated
What CIOs want
Reliabile software that functions as
promised
95
Software free from security
vulnerabilities and malicious code
70
Ease of Integration & Configuration
55
Software conforms to Requirements
& Industry Standards
32
Convenience & Ease of Use
27
Rich Feature Set
Other
11
0
0
https://www.cioexecutivecouncil.com October 11, 2006 Press Release
20
40
60
80
100
Percent
2
“Defacto” security requirements in NIST 800-53 do not explicitly
require developers to produce secure code
Technical
 Operational
– AC-2 Account Management
– CM-7 Least Functionality
– AC-3 Access Enforcement
– SI-3 Malicious Code Protection
– AC-4 Information Flow
Enforcement
– SI-10 Information Input Validation
Management
– RA-5 Vulnerability Scanning
3
Technical controls in NIST 800-53 contribute to application security
– AC-7 Unsuccessful Login
Attempts
– AC-10 Concurrent Session
Control
– IA-4 Identifier Management
– IA-5 Authenticator
Management
– SC-12 Cryptographic Key
Establishment and
Management
– IA-6 Authenticator Feedback
– SC-24 Fail in Known State
– IA-7 Cryptographic Module
Authentication
– SC-25 Thin Nodes
– AC-18 Wireless Access
– AC - 16 Security Attributes
– IA - 8 Identification and
Authentication (NonOrganizational Users)
– AC-19 Access Control for
Mobile Devices
– AC-17 Remote Access
– SC-2 Application Partitioning
– AC-20 Use of External
Information Systems
– SC-3 Security Function
Isolation
– AU-2 Auditable Events
– SC-4 Information in Shared
Resources
– AC-11 Session Lock
– AC-14 Permitted Actions
without Identification or
Authentication
– AU-3 Content of Audit Records
– AU-4 Audit Storage Capacity
– SC-7 Boundary Protection
– AU-9 Protection of Audit
Information
– SC-8 Transmission Integrity
– IA-3 Device Identification and
Authentication
– SC-11 Trusted Path
– SC-10 Network Disconnect
– SC-9 Transmission
Confidentiality
– SC-13 Use of Cryptography
– SC-28 Protection of
Information at Rest
– SC-23 Session Authenticity
– AC-5 Separation of Duties
– AC-6 Least Privilege
4
Operational controls in NIST 800-53 contribute to application
security
– SI-1 System and Information
Integrity Policy and Procedures
– SI-2 Flaw Remediation
– AT-2 Security Awareness
– AT-3 Security Training
– CM-3 Configuration Change
Control
– CM-4 Security Impact Analysis
– CM-5 Access Restrictions for
Change
– SI-4 Information System
Monitoring
– SI-6 Security Functionality
Verification
– SI-7 Software and Information
Integrity
– CA-5 Plan of Action and
Milestones
– RA-3 Risk Assessment
– SA-2 Allocation of Resources
– SA-3 Life Cycle Support
– SA-4 Acquisitions
– SA-6 Software Usage
Restrictions
– SA-10 Developer Configuration
Management
– SA-11 Developer Security
Testing
– SA-12 Supply Chain Protection
– SA-13 Trustworthiness
– PM-1 Information Security
Program Plan
Resources
– PM-6 Information Security
Measures of Performance
– PM-7 Enterprise Architecture
– PM-9 Risk Management
Strategy
– SA-8 Security Engineering
Principles
– CM-6 Configuration Settings
– CM- 8 Information System
Component Inventory
– SI-11 Error Handling
– SI-12 Information Output
Handling and Retention
– PM-3 Information Security
5
Key questions
Who asks developers to develop secure code?
Do developers know why they need to develop secure code?
What should developers do/not do to develop secure code?
Can developers develop secure code by themselves?
How do developers know they have developed secure code?
Why should developers care?
Next Steps?
6
Perspective on technology today
Technology is an integral
part of our lives
IT and Communications
products are assembled, built,
and transported by multiple
vendors around the world
7
Malicious actors are taking advantage of abundant opportunities to
tamper with and sabotage products …
What
commonalities
exist?
83% of victims were targets of
opportunity
92% of attacks were not highly
difficult
86% were discovered by a third
party
96% of breaches were avoidable
through simple or intermediate
controls
How do
breaches
occur?
50% utilized some form of
hacking
49% incorporated malware
(lower percentages included
physical attacks, privilege misuse,
and social tactics)
* Source – 2011 Verizon Data Breach Investigations Report
… ultimately compromising system integrity and operations
8
SwA requires multi-disciplinary collaboration
ISO/IEC 27000
Project
ISO/IEEE 15288
Management
System
Engineering
Information
Assurance
Communication
CMMI
Challenges
Common Criteria
Software
Acquisition
Software
Assurance
Software
Engineering
 Vocabulary
 Reserved Words
ISO/IEEE 12207
Test &
Evaluation
Safety and
Security
Information
Systems
Security
Engineering
 Priorities
 Perspective
 Experience
 Objectives
 Drivers
 Risks
Source: https://buildsecurityin.us-cert.gov/swa/procresrc.html
Without a common language we cannot communicate across disciplines
9
Acquirers of IT products and services trust that suppliers are
addressing cyber security without validating
Prepare for the
acquisition
Advertise the
acquisition and
select the supplier
Initiate an
agreement
Monitor the
agreement
Accept the
product or
service
Product Development and Maintenance
Requirements Management
Design/Develop
Test
47% do not perform acceptance testing of third-party code
30% do not use static analysis/manual code
27% do not practice secure design
19% do not carry out security requirement definition
46% use own development method, rather than SDL or CMM/CMMI
15% follow SDL
20% follow CMM/CMMI ®
61% had no special incentive program to get developers and testers to work together
More than 70% do not measure developers with security related metrics
ROI was greater for those who employed a coordinated, prescriptive approach
Source: Forrester, “State of Application Security,” January 2011
10
Key questions
Who asks developers to develop secure code?
Do developers know why they need to develop secure code?
What should developers do/not do to develop secure code?
Can developers develop secure code by themselves?
How do developers know they have developed secure code?
Why should developers care?
Next Steps?
11
Communication across organizational stakeholders is critical
to addressing SwA challenges
Development Project
Define Business Goals
Development Organization
DO 1 Establish the assurance
resources to achieve key
business objectives
DO 2 Establish the environment to
sustain the assurance
program within the
organization
Acquisition and Supplier
Management
AM 1 Select, manage, and use
effective suppliers and
third party applications
based upon their
assurance capabilities.
DP 1 Identify and manage risks
due to vulnerabilities
throughout the product and
system lifecycle
DP 2 Establish and maintain
assurance support from the
project
DP 3 Protect project and
organizational assets
Prioritize
funds and
manage risks
Enterprise Assurance
Support
ES 1 Establish and maintain
organizational culture
where assurance is an
integral part of achieving
the mission
ES 2 Establish and maintain the
ability to support
continued delivery of
assurance capabilities
ES 3 Monitor and improve
enterprise support to IT
assets
Development Engineering
DE 1 Establish assurance
requirements
DE 2 Create IT solutions with
integrated business
objectives and assurance
DE 3 Verify and Validate an
implementation for
assurance
Enable
Resilient
Technology
Sustained
environment to
achieve
business goals
through
technology
The Assurance PRM Is A Holistic Framework that connects CMMI and RMM to facilitate communication
https://buildsecurityin.us-cert.gov/swa/proself_assm.html
12
A majority of SwA best practices focus on developer-centric
audiences from a security point of view
Development Organization
DO 1 Establish the assurance
resources to achieve key
business objectives
DO 2 Establish the environment to
sustain the assurance
program within the
organization
Development Project
Development Engineering
DP 1 Identify and manage risks due to
vulnerabilities throughout the
product and system lifecycle
DP 2 Establish and maintain
assurance support from the
project
DP 3 Protect project and
organizational assets
DE 1 Establish assurance
requirements
DE 2 Create IT solutions with
integrated business objectives
and assurance
DE 3 Verify and Validate an
implementation for assurance
Assurance for CMMI ®
13
Key questions
Who asks developers to develop secure code?
Do developers know why they need to develop secure code?
What should developers do/not do to develop secure code?
Can developers develop secure code by themselves?
How do developers know they have developed secure code?
Why should developers care?
Next Steps?
14
100 apps written by 100 developers at 100 companies
We
Trust
83 apps have serious vulnerabilities
72 apps have cross site scripting
40 apps have SQL Injection
100 apps contain code of unknown
origin
90 apps use un-patched libraries with
known flaws
We
Hide
We
Blame
5 apps have had a scan or pentest
1 app has had a manual security code
review
0 apps provide any visibility into security
Why
1 company has a responsible appsec
program
1 developer has any security training
Adapted from: The Open Web Application Security Project ,Jeff Williams, Aspect Security, SWA Forum Sept 2010
15
Implementation lessons learned from some of the 1/100
companies that implement SwA successfully
 Who
 Why
– Secure development SMEs
– Customer pressure
– Developers
– Reaction to an incident
 What
 Why Not
– Measure progress (training, secure code reviews,
security change requests)
– Compliance drivers don’t exist
– Internal policy
– Secure software training is not given to
developers and architects
– Focus is on systems and networks
 When
– During product development process
 How
– During Leadership discussions
– Executive leadership commitment
– As part of development and acquisition reviews
– Translate ROI to project manager vocabulary
(cost, schedule, quality)
 Where
– Start small and build
– IT Development Organizations
– Use coding standards
– IT Acquisition Organizations
– Empower secure development to prevent a
product from moving to the next milestone
– IT Integrator Organizations
Courtesy of September 2010 SwA Panel SwA Practices
– Getting to Effectiveness in Implementation
– Avoid creating a new language
– Leverage what is already known
– Increase automation of menial tasks
16
Key questions
Who asks developers to develop secure code?
Do developers know why they need to develop secure code?
What should developers do/not do to develop secure code?
Can developers develop secure code by themselves?
How do developers know they have developed secure code?
Why should developers care?
Next Steps?
17
Measure, measure, and measure again
"The only man I know who
behaves sensibly is my tailor;
he takes my measurements
anew each time he sees me.
The rest go on with their old
measurements and expect
me to fit them."
- George Bernard Shaw
Source: www.CartoonStock.com
18
Efficiency/Effectiveness
Robust measurement does not happen overnight and requires
foundational capabilities in place to be effective
Goal
Metric Type
Add Essential
Monitoring
and Controls
Develop
Timeliness,
Accuracy,
Coverage
Prepare
Foundation
Develop Capability
Implementation
Metric
Effectiveness
Metric
Continuous Operational
Monitoring Supports
Response
--Maintain, and Improve
Security
Use Capability
Impact
Metric
Courtesy of Matt Coose, DHS
19
Critical success factor – long-term management commitment,
focus, and appropriate expectations
Problem
Solution
 Senior management has not expressed explicit
support for the program
 Obtain senior management commitment before
you start
 There is no long term funding and resource
commitment
 Work across the organization with the
stakeholders to make them a part of the solution
 There is a perception in the organization that
measurement and monitoring are temporary and
“will be done” at some point
 Iterate the program to measure critical things
 Security improvements are expected within a very
short time period and are expected to be easily
directly related to measurement and monitoring
 Program is expected to be perfectly planned and
executed as if the organization has done this a
million times and has the process down perfectly
 Turnover and changes in roles breaks up
continuity
 Structure the program to begin with quick wins to
continuously demonstrate increase in value
 Manage expectations continuously – explain that
the long term focus is critical
 Assign roles, train your responsible parties, and
communicate that continuity is key for success
 Emphasize positive reinforcement – if everyone is
failing, nobody will cooperate and the program will
fail
 Accountability for metrics is difficult to assign
20
Critical success factor – realistic and well thought out data
collection strategy
Problem
Solution
 Data sources not well known, well defined, and
therefore not considered
 Identify all available automated and manual
data sources
 Desired data is not obtainable
 Define data that you need and compare with
what available
 Automated data sources are dispersed
throughout the organization and data is difficult
to consolidate
 Authoritative data sources do not exist
 Data is collected inefficiently or incorrectly
 Collection deemed too difficult too early
 Difficulty or inability to capture historical data
 Create data collection strategy that would
balance the need for data with the current
state and plan for deliberately expanding data
sources and measures
 Define future changes to processes/tools or
requirement for new tools early in the process
and refresh as you learn more
 Use feasibility of data collection as one of the
criteria for metrics selection
 Train your data collectors and information
owners about what you need and then ask for
it repeatedly
21
Critical success factor – effective use of the measures to improve
security
Problem
Solution
 Metrics analysis is not prioritized according to
strategic goals, risks, ROI, or other explicit
criteria agreed upon by the organization
 Develop criteria for prioritizing and scoring
measures early in the program and reconsider
every time you expand the program
 Wrong types of measures are distributed to
wrong stakeholders
 Define who gets what data and reassess
periodically – ask your customers if it is still
useful
 Measures are collected for compliance and are
not used to improve security
 Metrics data is not used for risk-based decision
making
 If metrics are not used for decision making and
improvement, drop them
 Communicate, communicate, communicate
22
Security control measures
 Percent of new systems that have completed certification and accreditation (C&A) prior to their
implementation (NIST SP 800-53 Control: CA-6: Security Accreditation)
 Percent of employees who are authorized access to information systems only after they sign
an acknowledgement that they have read and understood rules of behavior (NIST SP 800-53
Controls – PL-4: Rules of Behavior and AC-2: Account Management)
 Percent of the agency’s information system budget devoted to information security (NIST SP
800-53 Controls – SA-2; Allocation of Resources)
Security Control Measures address compliance with the end state of the
system, but not the underlying processes, structures, and code
23
Measurement for secure code requires understanding code level
attributes …
24
Measurement for secure code involved understanding the
effectiveness of implemented processes
 Acquisition
– Number and percent of acquisition discussions that include SwA representative
– Number and percent of contracting officers who received training in the security
provisions of the FAR
– Percent of documented Supplier claims verified through testing, inspection, or other
methods
– Number and percent of relevant high impact vulnerabilities (CVEs) present in the system
 Testing
– Number and percent of tests that evaluate application response to misuse, abuse, or
threats
– Number and percent of tests that attempt to subvert execution or work around security
controls
– Percent of untested source code related to security controls and SwA requirements
25
How to apply…. success is simple
Get Management Support
Start Small
Measure Behavior
• Scope out measurement
• Obtain tangible support for
security measures
development and use at
every management level
• Maintain support through
regular reporting to
stakeholders, tailored to their
levels
–Address their goals
–Refine detail further up the
management chain
–Use Dashboards &
Reports to endure
• Expand your project/program cost, schedule,
quality, and growth measures to cover
security
• Start with a manageable, small set of
security measures adding more measures as
the project matures
• Leverage existing industry lists and select
applicable measures
• Use a framework to translate measures from
industry lists into your organization’s
approach
• Train data collectors to think in terms of
metrics
• Identify and measure best
and worst process and
practice behaviors as well
as results
• Create, display, and report
measures to influence
appropriate behavior
• Take advantage of
unintended consequences
produced by measurement
• Reuse measures where
possible
Incorporate security measures into your existing measurement activities
Source: http://www.psmsc.com/Downloads/TechnologyPapers/SwA%20Measurement%2010-08-08.pdf, accessed 4/10/09
26
Key questions
Who asks developers to develop secure code?
Do developers know why they need to develop secure code?
What should developers do/not do to develop secure code?
Can developers develop secure code by themselves?
How do developers know they have developed secure code?
Why should developers care?
Next Steps?
27
Business functions rely on accurate and reliable information from
technology that functions as intended (and only as intended)
Organization
Mission
Assets in Production
people
info
tech
Service
Service
Service
Mission
Mission
Mission
facilities
Adapted from: November 2009 SwA Forum-Evolution in SwA Processes Panel – David White, SEI
28
Potential impacts from threats to business functions can be
understood by communicating software level vulnerabilities
Are we being
attacked?
Compliance
OVAL
SCAP
Incident
Management
and Control
C
A
P
E
C
How do we prevent
this next time?
Resilience
Requirements
Management
Who is attacking and
what do they want?
Enterprise
Focus
Monitoring
B
P
M
N
Measurement
and Analysis
Controls
Vulnerability
Analysis and
Resolution
Applied to
Asset
Management
Are we at
risk?
Adapted from September 2010 SwA Forum,
CERT RMM for Assurance , Lisa Young, SEI
29
http://www.ruggedsoftware.org/
30
Key questions
Who asks developers to develop secure code?
Do developers know why they need to develop secure code?
What should developers do/not do to develop secure code?
Can developers develop secure code by themselves?
How do developers know they have developed secure code?
Why should developers care?
Next steps?
31
Cyber security and software assurance standard development
organization landscape
32
SC22 – Programming Languages, ISO/IEC TR 24772, Programming
Language Vulnerabilities
 Targets building software that is inherently less vulnerable through improving the programming
languages, or, at least, improve the usage of them in coding
 A catalog of 60+ issues that arise in coding when using any language and how those issues
may lead to security and safety vulnerabilities
 Cross-referenced to CWE
 Each discussion includes
– Description of the mechanism of failure
– Recommendations for programmers: How to avoid or mitigate the problem.
– Recommendations for standardizers: How to improve programming language specifications.
Courtesy of Jim Moore, MITRE
33
ISO/IEC 27036: Information technology – Security techniques –
Information Security for Supplier Relationships
 Scope: This international standard covers information security in relationships between
acquirers and suppliers to provide appropriate information security management for all parties.
In particular, it also includes management of information security risks related to these
relationships.
 The standard will be subdivided into the following parts:
– Part 1 – Overview and Concepts
– Part 2 – Common Requirements
– Part 3 – Guidelines for ICT Supply Chain
– Part 4 – Guidelines for Outsourcing
34
NIST IR 7622, Piloting Supply Chain Risk Management for
Federal Information Systems
 Initially based on DoD ICT SCRM Key Practices document and developed in close
collaboration with the industry
 Introduces the notion of supply chain players
– Acquirer - For this document, the acquirer is always a government agency (including those
agencies taking on the role of integrator).
– Integrator – A third-party organization that specializes in combining products/elements of
several suppliers to produce elements (information systems).
– Supplier – Third-party organization providing individual elements. Synonymous with vendor
and manufacturer; also applies to maintenance/disposal service providers
 Lays out pre-requisites of being able to address ICT SCRM challenge
 States specific practices that are consistent with DoD guidance and ISO frameworks
35
SAFECode (www.safecode.org)
 SAFECode is a global, industry-led effort to identify and
promote best practices for developing and delivering
more secure and reliable software, hardware and
services
 White papers
– Software Assurance: An Overview of Current
Industry Best Practices
– Fundamental Practices for Secure Software
Development
– Security Engineering Training: A Framework for
Corporate Training Programs on the Principles of
Secure Software Development
– Framework for Software Supply Chain Integrity
– Software Integrity Controls: An Assurance-Based
Approach to Minimizing Risks in the Software Supply
Chain
36
The Open Group
Trusted Technology Provider Framework (TTPF)
Purpose
Identify and gain consensus on common processes, techniques, methods, product and system
testing procedures, and language to describe and guide product development and supply chain
management practices that can mitigate vulnerabilities which could lead to exploitation and
malicious threats to product integrity.
Objectives
• Identify product assurance practices that should be expected from all commercial
technology vendors based on the baseline best practices of leading trusted commercial
technology suppliers
• Help establish expectations for global government and commercial customers when seeking
to identify a trusted technology supplier
• Leverage existing globally recognized information assurance practices and standards
• Share with commercial technology consumers secure manufacturing and trustworthy
technology supplier best practices
• Harmonize language used to describe best practices
Source: Source: September 28, 2010 SwA Forum, DoD Trusted Defense Systems, Ms. Kristen Baldwin, DDR&E/Systems Engineering
37
What’s next?
Continued collaboration to:
– Reach and enable developers
– Reach and enable executives
– Develop and promote resources for us by developers and executives
Participation in international standardization efforts
– SC7 TAG intersections through your SC7 TAG
– CS1/SC27
– IEEE representative to the SC7 TAG
– SC22
Participation through the SwA Working Groups and Forum
Stay Tuned …
38
Contact information
Nadya Bartol
Michele Moss
Senior Associate
Lead Associate
Booz Allen Hamilton Inc.
One Preserve Parkway
Rockville, MD 20852
Tel (301) 444-4114
bartol_nadya@bah.com
Booz Allen Hamilton Inc.
8283 Greensboro Drive
Mclean, VA 22102
Tel (703) 377-1254
moss_michele@bah.com
39