Cyber Metrics in the DoD
or
How Do We Know What We Don’t Know?
John S. Bay, Ph.D.
Executive Director
Things People Have Asked Me
• How much money should I spend this year on cyber
defense technologies?
• How many attacks has your firewall repelled this
month?
• If I only had a dollar to spend on cyber, where
should I spend it?
• Why is cyber research such a slog?
11/12/14
2
Answers
(which did not go over well)
• How much money have you got?
• We repelled all of them … except that one you read
about in the paper
• Spend your dollar on upgrades
• Cyber research is a slog because there is no physics
theory underlying it all, liker Maxwells’ Equations
or Newton’s Laws
11/12/14
3
But really … it DEPENDS
• The “threat” factor is common in cybersecurity,
but mostly not elsewhere
• … and it IS true that there is no useful PHYSICS
for the problem
11/12/14
4
DoD Taxonomy of Threats
Tier
Description
I
Practitioners who rely on others to develop the malicious code, delivery mechanisms, and execution
strategy (use known exploits).
II
Practitioners with a greater depth of experience, with the ability to develop their own tools (from
publically known vulnerabilities).
III
Practitioners who focus on the discovery and use of unknown malicious code, are adept at installing
user and kernel mode root kits, frequently use data mining tools, target corporate executives a
key users (government and industry) for the purpose of stealing personal and corporate data with the
expressed purpose of selling the information to other criminal elements.
IV
Criminal or state actors who are organized, highly technical, proficient, well funded professionals
working in teams to discover new vulnerabilities and develop exploits.
V
State actors who create vulnerabilities through an active program to “influence” commercial products
and services during design, development or manufacturing, or with the ability to impact
products while in the supply chain to enable exploitation of networks and systems of interest
VI
States with the ability to successfully execute full spectrum (cyber capabilities in combination with
all of their military and intelligence capabilities) operations to achieve a specific outcome in political,
military, economic, etc. domains and apply at scale.
From: Defense Science Board, Resilient Military Systems and the Advanced
Cyber Threat, January 2013
11/12/14
5
And The Corresponding Criticality
11/12/14
6
What Might the COSTS Be?
11/12/14
7
So Then, What to Measure?
• Qualitative
– Capabilities
– Missions lost
• Quantitative
– Performance
– Cost
• To achieve
• Not achieving
11/12/14
8
Capabilities and Maturity
11/12/14
9
Dashboard Approach
11/12/14
10
“Stoplight Chart” Assessments
See:
11/12/14
SPIDERS JCTD
11
Costs to Us
•
•
•
•
All vulnerabilities are bugs
All code has bugs
Bugs are expensive
Exploits are cheap  the “asymmetry” problem
11/12/14
12
Mission-Assurance Approach
11/12/14
•
Helps focus attention
•
Requires a “map” o the
mission
•
Implies a prioritization
on missions (something
loses)
•
Requires reconfigurable
systems and networks
•
Is not cheap
From: DUSD(I&E) Office, HANDBOOK
For SELF-ASSESSING SECURITY VULNERABILITIES & RISKS of
INDUSTRIAL CONTROL SYSTEMS
On DOD INSTALLATIONS, December 2012
13
Just Good Enough (Incremental)
Approach
•
How long would our red team take to penetrate the
system?
– An empirical measure, at best.
– Implies a canonical red team
prob(first
vulnerability is
discovered)
Bad code
Gamma
distribution?
Better code
time
11/12/14
14
The Accountability Approach
•
NIST 800-53 guidelines
•
The “did we do everything we know how to do” approach
From: NIST Special Publication 800-53, rev. 4, Security and Privacy Controls for
Federal Information Systems and Organizations, April 2013
11/12/14
15
Conclusions: Which is Best?
•
None of them. They service somewhat orthogonal
purposes.
– But they can provide apples-to-apples comparisons
•
Can they answer the Generals’ questions?
– No
– … except maybe the one about the firewall
– There is CERTAINLY no satisfactory “physics” to guide anybody
•
Cyber Metrics is still an extremely important and highpriority problem for OSD!
11/12/14
16