University of Southern California
Center for Systems and Software Engineering
Value Driven Security Threat Modeling
for Off The Shelf Software Systems
Yue Chen
USC Center for Systems and Software Engineering
Emails: {yuec, boehm, lshep}@usc.edu
Feb 14, 2007
CSSE Annual Research Review
Los Angeles, U.S.A.
©USC-CSSE
1
University of Southern California
Center for Systems and Software Engineering
Trends: Increasing Concerns on COTS
Vulnerability
Increasing Trend of COTS Based Applications
• Increasing COTS
usage
Data source: [Boehm et al
2003][Standish]
• Increasing
number of COTS
vulnerabilities
published
Data source: [CERT
Statistics]
©USC-CSSE
2
Introduction
University of Southern California
Center for Systems and Software Engineering
Prioritizing COTS Vulnerability is Important
• Significant Loss: CSI/FBI 2006 Survey reported a
total loss of $52,494,290 caused by security
incidents from 313 organizations
• Limited resources for security: 47% spend equal or
less than 2% in IT security
Data source: CSI/FBI Computer Crime and Security Survey 2006
• Challenge: How to spend limited amount of security
resources smartly?
©USC-CSSE
3
Introduction
University of Southern California
Center for Systems and Software Engineering
Current Practices
• System context neutral, static vulnerability rankings
are recommended by authority organizations such
as BugTraq/Symantec, CERT, ISS, NIST, Microsoft,
SANS
• Coarse granularity of rankings: three ~ four levels of
rankings (e.g. Critical, Important, Moderate, Low)
are used to differentiate 23,620 known COTS
vulnerabilities
• An Real-life Story
– USC Ticketing Office System (Source: USC-ISD)
• Decisions made based on [Butler, 2002]
– Best knowledge
– Individual experience
– Ad hoc
©USC-CSSE
4
Introduction
University of Southern California
Center for Systems and Software Engineering
Proposal: Threat Modeling framework based
on Attack Path analysis (T-MAP)
A novel threat modeling framework for COTS based
systems that is sensitive to system stakeholder value
context, dynamic, and tool-automated
Current Approaches:
COTS
Vulnerability
COTS
Vulnerability
Rankings
Value Neutral
Assessment
T- MAP:
COTS
Vulnerability
Value Neutral
Assessment
Scenario
Evaluation
COTS
Vulnerability
Rankings
Evaluation criteria
based on
stakeholder value
propositions
©USC-CSSE
5
Introduction
University of Southern California
Center for Systems and Software Engineering
Nature of The Problem
Attacking Paths
Permitted Ports
Unblocked vulnerabilities
Vulnerabilities impacting
confidentiality, availability,
integrity
Blocked vulnerabilities
Firewall Wrapper
e.g. Windows Server
2003
e.g. SQL Server
2000
Software Applications, COTS
e.g. Web Server
e.g. IIS 6.0
e.g. CRM Server
IT Infrastructure
e.g. Regulatory
Productivity
Org. Values
©USC-CSSE
Reputation
T-MAP 6
Framework
University of Southern California
Center for Systems and Software Engineering
Castle Defense Analog
Measure the security of a castle by the value of
treasures in the castle, the number of holes on the
walls, as well as the size of the holes.
©USC-CSSE
T-MAP 7
Framework
University of Southern California
Center for Systems and Software Engineering
T-MAP Framework
• Step 1: Identify key stakeholders and value propositions (the treasures in
the castle);
• Step 2: Establish a set of security evaluation criteria based on
stakeholder value propositions;
• Step 3: Use tool to enumerate and analyze attack paths based on a
comprehensive COTS vulnerability database containing 23,620
vulnerability information (the holes);
• Step 4: Evaluate the severity of each scenario in terms of numeric ratings
against the evaluation criteria established in Step 2 (the size of the
holes);
• Step 5: The security threat of each vulnerability is quantified with the total
severity ratings of all attack paths that are relevant to this vulnerability;
• Step 6: System total threat is quantified with the total severity ratings of
all attack paths;
[Note] Step 3 to 6 are tool automated by the Tiramisu Tool
©USC-CSSE
T-MAP 8
Framework
University of Southern California
Center for Systems and Software Engineering
Step 1-2: Evaluate Security against Criteria
based on Stakeholder/Values
• Evaluate the severity of security hazard scenarios
against stakeholder/value impacts
• Involves both qualitative and quantitative criteria
• Technical approach: Figure of Merits and Analytical
Hierarchy Process (AHP)
– A convenient tool to determine the priority of alternatives in terms
of weight through pair-wise comparison
– Invented by Dr. Saaty at Business School of Univ. of Pennsylvania
in 1970s [Saaty, 1980]
– Recommended by Dr. Bodin, Dr. Gordon et al for security
investment evaluation [Bodin et al, 2005]
©USC-CSSE
T-MAP 9
Framework
University of Southern California
Center for Systems and Software Engineering
Using AHP Determine the Weights
– An Example
Example (from USC ISD Server X Case Study)
Weights derived
through AHP
pair-wise comparisons
Possible
Breach
Scenarios
Value Centric
Criteria
©USC-CSSE
T-MAP 10
Framework
University of Southern California
Center for Systems and Software Engineering
Step 3-4: Enumerate Attack Scenarios
• Enumerate the scenarios how an attacker can compromise
stakeholder values through COTS system vulnerabilities
• Establish Structured Attack Graph based on a comprehensive
COTS vulnerability database involves 23,620 known vulnerabilities
reside in 31,713 COTS software
©USC-CSSE
T-MAP 11
Framework
University of Southern California
Center for Systems and Software Engineering
Step 5: Vulnerability Technical
Severity Evaluation
• Vulnerability Technical
Severity Attributes
– Impact on confidentiality,
integrity and/or availability
– Remotely exploitable
– Require valid user account
on victim host
– Needs user activities
Based on an emerging national
standard [CVSS] by NIST
©USC-CSSE
T-MAP 12
Framework
University of Southern California
Center for Systems and Software Engineering
Step 6 T-MAP Severity Rating System
• Severity Weight of Attack Path P:
• Overall Security Threat Score of COTS System G:
• ThreatKey of elements in Attack Graph:
• Effectiveness of Security Practice:
©USC-CSSE
T-MAP 13
Framework
University of Southern California
Center for Systems and Software Engineering
Security Investment Effectiveness
Estimation
• How much security threats can be avoided by implementing
Firewall, Software hardening (patching), user account control, or
file system encryption?
• Results as well depends on the total value of the protected system
* Case study results estimated by professional security manager at USC-ITS
T-MAP 14
©USC-CSSE
Framework
University of Southern California
Center for Systems and Software Engineering
Tool Implementation – Project Code:
Tiramisu
High-level software architecture
How it works?
NIST
FrSIRT
Microsoft
CERT
SANS
Symantec
BugTraq
Vulnerability Information
Scrawling Engine
T-MAP
Tiramisu
Front End
Vulnerability DB
Tiramisu Tool
©USC-CSSE
T-MAP 15
Implementation
University of Southern California
Center for Systems and Software Engineering
Security Manager’s Ranking vs.
Tiramisu Ranking
©USC-CSSE
T-MAP 16
Framework
University of Southern California
Center for Systems and Software Engineering
Security Economics in Patching
• Prioritize COTS Based System vulnerabilities under business
context
– “20% percent of vulnerabilities causes 80% of the security
risks”, T-MAP tells what are the 20%
• Rational: Prioritize vulnerabilities with its ThreatKey;
• Example screenshot:
©USC-CSSE
17
University of Southern California
Center for Systems and Software Engineering
COTS Security Economics –
Finding Sweet Spots
• Sweet spot to invest in
• Economic curve of security
security
patching
• Also driven by the total
(from USC Server X case study)
value of system
(from USC Server X case study)
Sweet spots to invest
©USC-CSSE
18
University of Southern California
Center for Systems and Software Engineering
Limitations
• Only sensitive to known COTS Vulnerabilities
– Empirical study by Arora shows that the average attacks per host
per day jumped from 0.31 to 5.45 after vulnerability get published
• Not sensitive to nuance in local system
configurations
– Disabled services
– Services running on different privileges, etc.
• Only cover “one-step-attacks” that exploiting COTS
vulnerabilities
• Depends on comprehensive vulnerability database
– Our database: 23,260 vulnerability published from 1999-2006 that
resides in 31,313 COTS software
• Cannot effectively address attacks such as Phishing
©USC-CSSE
T-MAP 19
Framework
University of Southern California
Center for Systems and Software Engineering
Hypothesis
Hypothesis #1 *: For given COTS based system whose confidentiality, integrity
and availability have different priorities to the stakeholder values, the
accuracy of T-MAP results, measured by the Inaccuracy, the ratio of the
number of clashes between vulnerability priorities and stakeholder value
priorities and the total number of comparisons, will not make any difference
comparing to the existing leading approaches (say, the average level of at
least three leading approaches, if rating data available).
Clash definition: for two technically identical vulnerabilities VA and VB, given
the stakeholder value impact severity of VA is higher than VB, if a
prioritizing system assign VA less or equal priorities as VB, it is counted as
a clash for this prioritizing system.
* Hypothesis #1 is a null hypothesis. We aim at disprove them through case
studies and tool demos.
Clash Counting Illustration
Vuln.
VA
©USC-CSSE
VB
- - - - - - X -
-
Research
20
Hypothesis
University of Southern California
Center for Systems and Software Engineering
Inaccuracy Comparison
• Inaccuracy *: the ratio of the number of clashes between
vulnerability priorities and stakeholder value priorities and
the total number of comparisons
* Inaccuracy is a numerical value between 0~1, the
larger value indicates the less accuracy
For the USC ISD case study, the security manager
prioritized 8 vulnerabilities, which translates to 28
comparisons,
where 28 = 1+2+… + 7
# of Clashes
Inaccuracy
T-MAP
2
0.071
CVSS
6
0.214
ISS
9
0.321
©USC-CSSE
Hypothesis
21
Validation
University of Southern California
Center for Systems and Software Engineering
Contributions
• A framework to model CBS security
– A COTS security evaluation framework that
captures stakeholder value propositions
– Distill the potential impacts of thousands of
vulnerabilities into management friendly
numbers at a high-level
– Tool automated
• Has the potential to pro-actively attack
security in early life-cycle, instead of taking
whatever as is reactively after system built
©USC-CSSE
Contribution
&
22
Next Steps
University of Southern California
Center for Systems and Software Engineering
Discussions and Future Work
©USC-CSSE
23
University of Southern California
Center for Systems and Software Engineering
Thanks!
©USC-CSSE
24