Are Enterprise Security Risk
Metrics Really Needed?
Robert Marchant Ph.D.
Technical Fellow
Sotera Defense Solutions
•
Information Systems Executives and Program Managers Need Security
Metrics to Establish an Effective Security Budget.
•
•
[Evans 2004] Evans, Karen, Testimony before the Committee on Government Reform,
Subcommittee onTechnology, Information Policy, Intergovernmental Relations, and the
Census, 16 March 2004.
IRC recognizes Enterprise Security Metrics as a hard problem
•
INFOSEC Research Council 2005 Hard Problems List- number eight
•
The Goal is Metrics at Least as Good as What is Used for Program Risk
Management.
•
The problem is real, but quantitative is a weak hypothesis.
•
Quantities are based on qualitative data
•
The problem and the methodology (framework) are organizationally agnostic.
•
The two most often used IT risk frameworks are virtually interchangeable.
•
[Kuligowski 2009] Kuligowski, C., Comparison of IT Security Standards. Masters Thesis,
http://www.federalcybersecurity.org/CourseFiles/WhitePapers/ISOvNIST.pdf/
•
The IRC states that organizations must make cost/benefits
decision based on data that is (at best) poorly quantified.
• These unqualified decisions are often based on short term,
poorly associated metrics that often leads to decision that
result in poor, long term decision.
• According the IRC, “One of the most insidious threats to
security metrics lies in the metrics themselves. The mere
existence of a metric may encourage its purveyors to over
endow the significance of the metric. A common risk is that
analyses may be based on spurious assumptions, inadequate
models, and flawed tools, and that the metrics themselves are
inherently incomplete --- often a one-dimensional projection of
a multidimensional situation. Furthermore, a combination of
metrics in the small (e.g., regarding specific attributes of
specific components) typically do not compose into metrics in
the large (e.g., regarding the enterprise as a whole).”
3
•
DMZ vs Cross Domain Guard
•
Network Interface rules
•
•
•
•
CDG are monolithic (single
probability)
DMZ is layered with multiple
protocol breaks (conditional
dependence – Def in Depth)
•
•
Keep the good stuff in
Keep the bad stuff out
Don’t allow covert channels
Java 1.7
•
Situational probability
4
Systems engineering lifecycle all have
iterative models, have feedback loops,
and all use some form of control gates.
Program risk assessment is performed
throughout the life cycle. Risk metrics
are normalized (dollarized)
Typical Risk Management Framework
6
FIPS 199/SP 800-60
CATEGORIZE
Information System
SP 800-37/SP 800-53A
MONITOR
Security Controls
Define criticality/sensitivity of
information system according
to potential worst-case,
adverse impact to
mission/business.
Continuously track changes to the
information system that may
affect security controls and
reassess control effectiveness.
SP 800-37
Security Life Cycle
Security Controls
SP 800-70
IMPLEMENT
Security Controls
Information System
SP 800-53A
ASSESS
Security Controls
7
SELECT
Select baseline security controls;
apply tailoring guidance and
supplement controls as needed
base on risk assessment
AUTHORIZE
Determine risk to organizational
operations and assets, individuals,
other organizations, and the
Nation; if acceptable, authorize
operation.
FIPS 200/SP 800-18
SP 800-30/SP 800-53
Determine security control
effectiveness (i.e., controls
implemented correctly, operating
as intended, meeting security for
information systems).
Implement security controls
within enterprise architecture
using sound systems
engineering practices; apply
security configuration settings
SDLC to RMF comparison
8
NIST Risk Management Framework
Starting Point
CATEGORIZE
Information System
MONITOR
Define criticality/sensitivity of
information system according to
potential worst-case, adverse
impact to mission/business.
SELECT
Security Controls
Security Controls
Continuously track changes to the
information system that may affect
security controls and reassess
control effectiveness.
Select baseline security controls;
apply tailoring guidance and
supplement controls as needed
base on risk assessment
AUTHORIZE
Information System
Determine risk to organizational
operations and assets, individuals,
other organizations, and the Nation; if
acceptable, authorize operation.
Security Engineering
and IT Risk Assessment
Performed Here
ASSESS
Security Controls
Determine security control
effectiveness (i.e., controls
implemented correctly, operating as
intended, meeting security for
information systems).
IMPLEMENT
Security Controls
Implement security controls within
enterprise architecture using
sound systems engineering
practices; apply security
configuration settings
The Quantitative Measure
•
Risk = Asset Value * Probability * Impact
•
•
•
•
•
•
•
Where
Asset value is in $s (SME based)
Probability based on threat analysis (SME based)
Impact (Vulnerability DB based or SME)
The sum of the risk is the total enterprise metric
Security Engineers evaluate the threat and
modify/influence architecture/design to minimize.
Maintaining these metrics at the enterprise level would
require a normalized taxonomy (or ontology)
•
•
Measure the probability?
Requires experts and security engineers
Quantitative Metrics Are Expensive
•
Better to minimize by lowering probability
•
•
Enterprises change
•
•
•
•
•
•
New capabilities
Assimilation
Planned upgrades
Response to new threats
Normalized taxonomy (ontology)?
SMEs can be assembled as needed!
•
•
[Coles-Kemp 2009] Coles-Kemp, L. (2009). The Effect of Organisational
Structure and Culture on Information Security Risk Processes Administrative
Science Quarterly, 17(1), 1- 25.
But make sure they are SMEs!!!!
Budget should be focused on maintaining security controls and
ensuring patches are current
•
Can be empirically measured
Questions?
Backup Slides
•
Head of Agency (Element Head, Chief Executive, e.g. Director National
Security Agency or Secretary of Defense): The executive with the ultimate
responsibility for mission accomplishment and execution of business functions.
•
Risk Executive (an individual or a function): The Risk Executive function may
be fulfilled by an individual or a group within an organization. The Risk Executive
(organization or individual) is primarily a source of expertise and consultation and
is usually a department or group (e.g. the technical security group or Cyber
Security Group).
•
Chief Information Officer (CIO): The CIO ensures that information systems are
acquired and information resources are managed in a manner consistent with
laws, Executive Orders, directives, policies, regulations, as well as priorities
established by the Element Head.
•
Senior Agency Information Security Officer (SAISO)/Chief Information
Security Officer (CISO): A SAISO or CISO executes the CIO’s responsibilities
under the Federal Information Security Management Act (FISMA) of 2002 and
serves as the CIO’s liaison to the organizations Authorization Official. It is this
individual who will aggregate all the organizations systems and programs FISMA
reporting into a single agency report to the OMB.
•
Authorizing Official (AO): An AO is an agency or element CIO or executive
of sufficient seniority to execute the decision-making and approval
responsibilities for information systems authorizations to operate (called and
ATO) on behalf of the Element Head. The AO assumes responsibility for
operating an IS at an acceptable level of risk to the organization.
•
Delegated Authorizing Official (DAO): A DAO is delegated authority by an
AO to carry out the same activities as an AO (e.g., authorize system
operations).
•
Security Control Assessor (SCA): An SCA (sometimes called a certifier) is
responsible for performing the evaluation (Asses Security Controls phase) of
the security-controls and features of an IS and determining the degree to
which the system meets its security requirements.
•
Common Control Provider (CCP): A CCP is responsible for all aspects of
providing common controls (i.e. the security controls from SP 800-53,
modification to the SP 800-53 recommended controls and any custom
controls augmenting SP 800-53). Organizations may have multiple CCPs.
•
Information Owner/Steward: An Information Owner/Steward is an
organization official who “owns the data”. The IO has statutory,
management, or operational authority for specific information and is
responsible for establishing the policies and procedures governing its
generation, collection, processing, dissemination, and disposal.
•
Mission/Business Owner (MBO): An MBO has operational responsibility
for the mission or business process supported by the mission/business
segment or the information system. The MBO is a key participant/stakeholder
regarding system life-cycle decisions.
•
Information System Owner (ISO)/Program Manager (PM): An ISO (aka
PM) is responsible for the overall procurement, development, integration,
modification, operation, maintenance, and disposal of an information system
(as well as the system components), to include development and provision of
the stem’s Security Plan (SSP).
•
Information System Security Engineer (ISSE): An ISSE ensures that
information-security requirements are effectively implemented throughout
the security architecting, design, development, configuration, and
implementation processes. The ISSE coordinates his/her security-related
activities with ISOs, ISSOs/ISSMs, and CCPs. The ISSE also provides
the definition, design, development, and deployment support to
development systems as part of the system under developments systems
engineering activity.
•
Information System Security Officer (ISSO)/Information System
Security Manager (ISSM): An ISSM or ISSO is responsible for
maintaining the day-to-day security posture and continuous monitoring of
a system.