Quantitative
Risk Analysis
Sanjay Goel
University at Albany, SUNY
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
1
Course Outline
> Unit 1: What is a Security Assessment?
– Definitions and Nomenclature
Unit 2: What kinds of threats exist?
– Malicious Threats (Viruses & Worms) and Unintentional Threats
Unit 3: What kinds of threats exist? (cont’d)
– Malicious Threats (Spoofing, Session Hijacking, Miscellaneous)
Unit 4: How to perform security assessment?
– Risk Analysis: Qualitative Risk Analysis
Unit 5: Remediation of risks?
– Risk Analysis: Quantitative Risk Analysis
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
2
Quantitative Risk Analysis
Outline for this unit
Module 1: Quantitative Risk Analysis and ALE
Module 2: Risk Aggregation
Module 3: Case Study
Module 4: Cost Benefit Analysis and Regression Testing
Module 5: Modeling Uncertainties
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
3
Module 1
Quantitative Risk Analysis and ALE
Quantitative Risk Analysis and ALE
Outline
•
•
•
•
•
•
•
What is Risk Analysis?
What is Quantitative Risk Analysis?
What are the steps involved?
How to determine the Likelihood of Exploitation?
How to determine Risk Exposure?
How to compute Annual Loss Expectancy (ALE)?
Examples
– Gym Locker
– Hard Drive Failure
– Virus Attack
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
5
Quantitative Risk Analysis and ALE
Risk Analysis Definition
• Risk analysis involves the identification and
assessment of the levels of risks calculated from the
known values of assets and the levels of threats to,
and vulnerabilities of, those assets.
• It involves the interaction of the following elements:
–
–
–
–
–
–
Assets
Vulnerabilities
Threats
Impacts
Likelihoods
Controls
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
6
Quantitative Risk Analysis and ALE
Risk Analysis Concept Map
• Threats exploit system vulnerabilities which expose system assets.
• Security controls protect against threats by meeting security
requirements established on the basis of asset values.
Source: Australian Standard Handbook of Information Security Risk Management – HB231-2000
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
7
Quantitative Risk Analysis and ALE
Quantitative Risk Analysis
• Quantitative risk analysis methods are based on
statistical data and compute numerical values of risk
• By quantifying risk, we can justify the benefits of
spending money to implement controls.
• It involves three steps
– Estimation of individual risks
– Aggregation of risks
– Identification of controls to mitigate risk
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
8
Quantitative Risk Analysis and ALE
Risk Analysis Steps
Security risks can be analyzed by the following steps:
• Identify and determine the value of assets
• Determine vulnerabilities
• Estimate likelihood of exploitation
– Compute frequency of each attack (with & w/o controls) using
statistical data
• Compute Annualized Loss Expectancy
– Compute exposure of each asset given frequency of attacks
• Survey applicable controls and their costs
• Perform a cost-benefit analysis
– Compare exposure with controls and without controls to determine the
optimum control
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
9
Quantitative Risk Analysis and ALE
Determining Assets and Vulnerabilities
• Identification of Assets and Vulnerabilities is the
same for both Qualitative and Quantitative Risk
Analysis
• The differences in both of these is in terms of
valuation:
– Qualitative Risk Analysis is more subjective and relative
– Quantitative Risk Analysis is based on actual numerical
costs and impacts.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
10
Quantitative Risk Analysis and ALE
Determine Likelihood of Exploitation
• Likelihood relates to the stringency of existing controls
– i.e. likelihood that someone or something will evade controls
• Several approaches to computing probability of an event
– classical, frequency and subjective
• Probabilities hard to compute using classical methods
– Frequency can be computed by tracking failures that result in security
breaches or create new vulnerabilities can be identified
– e.g. operating systems can track hardware failures, failed login attempts,
changes in the sizes of data files, etc.
• Difficult to obtain frequency of attacks using statistical
data.Why?
– Data is difficult to obtain & often inaccurate
• If automatic tracking is not feasible, expert judgment is used to
determine frequency
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
11
Quantitative Risk Analysis and ALE
Approaches
• Delphi Approach
– Probability in terms of integers (e.g. 1-10)
• Normalized
– Probability in between 0 (not possible) and 1
(certain)
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
12
Quantitative Risk Analysis and ALE
Delphi Approach
Frequency
Ratings
More than once a day
10
Once a day
9
Once every three days
8
Once a week
7
Once in two weeks
6
Once a month
5
Once every four months
4
Once a year
3
Once every three years
2
Less than once in three years 1
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
• Subjective probability
technique originally
devised to deal with
public policy decisions
• Assumes experts can
make informed decisions
• Results from several
experts analyzed
• Estimates are revised
until consensus is
reached among experts
13
Quantitative Risk Analysis and ALE
Risk Exposure
• Risk is usually measured as $ per annum and is
quantified by risk exposure.
– ALE (Annual Loss Expectancy, expressed as: $/year)
• If an event is associated with a loss
– LOSS = RISK IMPACT ($)
• The probability of an occurrence is in the range
of:
– 0 (not possible) and 1 (certain)
• Quantifying the effects of a risk by multiplying risk
impact by risk probability yields risk exposure.
– RISK EXPOSURE = RISK IMPACT x RISK
PROBABILITY
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
14
Quantitative Risk Analysis and ALE
Intangible Assets
• Incorporating intangible assets within Quantitative
Risk Analysis is difficult as it is hard to put a price
on things such as trust, reputation, or human life.
• However, it is necessary to put an as accurate a
value as possible when factoring these assets
within risk analysis as they may be even more
important than tangible assets.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
15
Quantitative Risk Analysis and ALE
Computing ALE
• Single Loss Expectancy: Loss to an asset if event occurs
– Value of the lost asset = Ci
– Impact on the Asset (if event occurs) = Pi
– SLE = Ci * Pi
• Annualized Rate of Occurrence (ARO) characterizes, on
an annualized basis, the frequency with which a threat is
expected to occur.
• Annualized Loss Expectancy (ALE) computes risk using
the probability of an event occurring over one year.
• Formulation
– ALE = (SLE)(ARO)
•
Source: Handbook of Information Security Management, Micki Krause and Harold F. Tipton
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
16
Quantitative Risk Analysis and ALE
Example #1: Gym Locker
Scenario: There is a gym locker used by its members
to store clothes and other valuables. The lockers
cannot be locked, but locks can be purchased.
You need to determine:
1) Risk exposure for gym members
2) Controls to reduce risk
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
17
Quantitative Risk Analysis and ALE
Example #1: Gym Locker, cont’d.
• Identify assets and determine value
–
–
–
–
–
–
–
–
Clothes
Wallet
Glasses
Sports equipment
Driver’s license
Car keys
House keys
Tapes and walkman
– Total Loss/week:
$50
$100
$100
$30
$20
$100
$60
$40
____
$500
• Find vulnerability
–
–
–
–
Theft
Accidental loss
Disclosure of information (e.g. read wallet)
Vandalism
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
18
Quantitative Risk Analysis and ALE
Example #1: Gym Locker, cont’d.
• Estimate likelihood of exploitation
–
–
–
–
–
10 (more than once a day)
9(once a day)
7 (once a week)
6 (once every two weeks)
5 (once a month)
–
–
–
–
4 (once every four months)
3 (once a year)
2 (once every three years)
1 (less than once every 3 years)
• For theft: estimated likelihood is 7
• Figure annual loss:
– ~$500 worth of loss each week, ~52 weeks in a year
– ~$26,000 loss per year
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
19
Quantitative Risk Analysis and ALE
Example #1: Gym Locker, cont’d.
• Determine cost of added security
– New lock $5
– Replacement for lost key $10
– On average members lose one key twice a month (24 times per year)
• Estimate likelihood of exploitation under added security
– The new likelihood of theft could be estimated at a 4.
• Cost Benefit Analysis
– Revised Losses (including cost of controls) =
(500 * 4) + (15*24) = 2360
– Net savings = 26000 – 2360 = 23640
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
20
Quantitative Risk Analysis and ALE
Example #2: Hard Drive Failure
• The chance of your hard drive failing is once every three years
– Probability = 1/3
• Intrinsic Cost
– $300 to buy new disk
• Hours of effort to reload OS and software
– 10 hours
• Hours to re-key assignments from last backup
– 4 hours
• Pay per hour of effort
– $10.00 per hour
• Total loss (risk impact)
– $300 + 10 x (10+4) = $440
• Annual Loss Expectancy (pa = per annum)
– (440 x 1/3)$pa = $147 pa
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
21
Quantitative Risk Analysis and ALE
Example #3: Virus Attack
• Situation: Virus Attack on same system
– You frequently swap files with other people, but have no
anti-virus software running.
– Assume an attack every 6 months (Probability = 2 per year)
– No need to buy a new disk
– Rebuild effort (10 + 4) hours
– Total loss = $10 x (10 + 4) = $140
– ALE = ($140 x 2) $pa = $280 pa
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
22
Quantitative Risk Analysis and ALE
Questions 1 and 2
1) Why is it important to quantify risk?
2) Give the definitions for:
a.
b.
c.
Single Loss Expectancy
Annualized Rate of Occurrence
Annual Loss Expectancy
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
23
Quantitative Risk Analysis and ALE
Question 3
3) For this situation:
a.
Same system as examples 2 and 3
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
24
Module 2
Risk Aggregation
Risk Aggregation
Outline
•
•
•
How do you determine risk posture?
What is this risk aggregation model?
Matrices
– Asset/Vulnerability
– Vulnerability/Threat
– Threat/Control
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
26
Risk Aggregation
Risk Posture
• Individual risks aggregated = Total risk posture
– True comparison of relative risks of different organizations
• Mathematical approach for aggregation provided
– Methodology standardized
– Data needs to be customized to organization
• Controls can reduce the cost of exposure
– Need to determine optimum controls for organization
– Methodology for determining controls shown next slide
• Analysis should be undertaken to see the impact of
new projects on security
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
27
Risk Aggregation
Model
• Let:
A be a vector of loss of an asset where al is the lth asset, s.t., 0 < l < L
V be a vector of vulnerabilities where vk is the kth vulnerability, s.t., 0 < k < K
T be a vector of threats where tj is the jth asset, s.t., 0 < j < J
C be the vector of vulnerabilities where ci is the ith control, s.t., 0 < i < I
Also Mα be the matrix that defines the impact of vulnerabilities (breach in
security) on assets, where, αkl is the impact of kth vulnerability on the lth asset
– Also Mβ be the matrix that defines the impact of threats on the vulnerabilities,
where, βjk is the impact of jth threat on kth vulnerability
– Also Mγ be the matrix that defines the impact of a controls (breach in security)
on the threats, where, γij is the impact of ith control on the jth threat
–
–
–
–
–
The notation is graphically explained in the next few slides
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
28
Risk Aggregation
Model, cont’d.
V (Vulnerabilities)
A (Assets)
•
akl
• Data Collection:
L
K
Where akl is the Impact of
vulnerability k on given asset l.
–
– Primary Data from
corporations that track financial
losses due to different attacks
– Secondary Data from the
reports of financial loss from
organizations like CERT,
CSI/FBI and AIG
– Data specific to a corporation,
could perhaps be classified into
different groups of companies
i.e. fraction of the asset value
that will be lost if the
vulnerability is exploited
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
29
Risk Aggregation
Model, cont’d.
T (Threats)
V (Vulnerabilities)
bjk
• Data Collection:
K
J
– Threat data and frequency of
threats is information that is
routinely collected in CERT and
other such agencies.
– Log data and collected data from
the organization itself can be
another source of information
– Data can also be collected via use
of automated monitoring tools
bjk is the probability that threat j
will exploit vulnerability k
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
30
Risk Aggregation
Model, cont’d.
C (Controls)
T (Threats)
gij
• Data Collection:
J
– Approximate control data can be
procured from various industry
vendors who have done extensive
testing with tools.
– Other sources of data can be
independent agencies which do
analysis on tools.
I
gij is the fraction by which controls
reduce the frequency of a threat
exploiting a vulnerability
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
31
Risk Aggregation
Model, cont’d.
Then losses if no control exist
J
K L
R    t j  b jk  a kl  al
j 1k 1l 1
Then losses if controls exist
J
K L
I
I
R*     (  ij )  b jk  a kl  al   Ci
j 1k 1l 1 i 1
 ij  (1  g ij )
i 1
= sum
 = product
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
32
Risk Aggregation
Optimization
If ζ is the maximum allocated budget for controls the
optimization problem can be formulated as:
J
K L
I
Minimize : R*    (  ij )  b jk  a kl  al
j 1k 1l 1 i 1
I
where,  Ci  
i 1
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
33
Risk Aggregation
Question 1
1)
How would you collect data for the following:
a.
Assets and Values
b.
Potential Threats
c.
Exploitable Vulnerabilities
d.
Possible Controls
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
34
Module 3
Case Study
Case Study
Outline
•
•
What is the case about?
What would fit into the categories of:
–
–
–
–
•
Assets
Vulnerabilities
Threats
Controls
Filling in the matrices
– Asset/Vulnerability
– Vulnerability/Threat
– Threat/Control
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
36
Case Study
Example
• Use the information that you have learned in the lecture in
the following case study of a government organization.
• Remember these key steps for determining ALE
– Identify and determine the value of assets
– Determine vulnerabilities
– Estimate likelihood of exploitation
– Compute ALE
– Survey applicable controls and their costs
– Perform a cost-benefit analysis
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
37
Case Study
Case
An organization delivers service throughout New York State. As part of the planning process to prepare the annual
budget, the Commissioner has asked the Information Technology Director to perform a risk analysis to determine
the organization’s vulnerability to threats against its information assets, and to determine the appropriate level of
expenditures to protect against these vulnerabilities.
The organization consists of 4,000 employees working in 200 locations, which are organized into 10 regions. The
average rate of pay for the employees is $20/hr. Cost benefit analysis has been done on the IT resource
deployment, and the current structure is the most beneficial to the organization, so all security recommendations
should be based on the current asset deployment.
Each of the 200 locations has approximately 20 employees using an equal number of desktop and laptop
computers for their fieldwork. These computers are used to collect information related to the people served by the
organization, including personally identifying information. Half of each employee’s time is spent collecting
information from the clients using shared laptop computers, and half is spent processing the client information at
the field office using desktop computers. Replacement cost for the laptops is $2,500 and for the desktop is $1,500.
Each of the 10 regions has a network server, which stores all of the work activities of the employees in that region.
Each server will cost $30,000 to replace, plus 80 hours of staff time. Each incident involving a server costs the
organization approximately $1,600 in IT staff resources for recovery. Each incident where financial records or
personal information is compromised costs the organization $15,000 in lawyers time and settlement payouts.
Assume that the total assets of the organization are worth 10 million dollars.
The organization has begun charging fees for the public records it collects. This information is sold from the
organization website at headquarters, via credit card transactions. All of the regional computers are linked to the
headquarters via an internal network, and the headquarters has one connection to the Internet. The headquarters
servers query the regional servers to fulfill the transactions. The fees collected are approximately $10,000 per day
distributed equally from each region, and the transactions are uniformly spread out over a 24 hour period.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
38
Case Study
Example- Assets (Tangible)
• Transaction Revenue- amount of profit from transactions
• Data- client information
• Laptops- shared, used for collecting information
• Desktops- shared, used for processing client information
• Regional Servers- stores all work activities of employees in
region
• HQ Server- query regional servers to fulfill transactions
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
39
Case Study
Example- Asset Valuations (Cost per Day)
Transaction Revenue
$10,000 per day
Data (Liability)
$10 million (total assets of organization)
Laptops
½ x 200 (locations) x 20 (employees) x
$2,500 (laptop cost) = $5,000,000
Desktops
½ x 200 (locations) x 20 (employees) x
$1,500 (desktop cost) = $3,000,000
Regional Servers
$30,000 (server cost)x 10 (regions) +
80 (hours) x $20 (pay rate) x 10 (regions)+
$10,000 (transaction revenue) = $326,000
HQ Server
$10,000 (transaction revenue) +
$100,000 (cost of HQ server) +
80 (hours) x $20 (pay rate) x 10 (regions) = $126,000
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
40
Case Study
Example- Vulnerabilities
• Vulnerabilities are weaknesses that can be exploited
• Vulnerabilities
– Laptop Computers
– Desktop Computers
– Regional Servers
– HQ server
– Network Infrastructure
– Software
• Computers and Servers are vulnerable to network attacks
such as viruses/worms, intrusion & hardware failures
• Laptops are especially vulnerable to theft
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
41
Case Study
Example- Threats
• Threats are malicious & benign events that can exploit
vulnerabilities
• Several Threats exist
–
–
–
–
–
–
–
Hardware Failure
Software Failure
Theft
Denial of Service
Viruses/Worms
Insider Attacks
Intrusion and Theft of Information
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
42
Case Study
Example- Controls
•Intrusion detection and firewall upgrades on HQ Server
– mitigate HQ server failure and recovery
•Anti-Virus Software
– mitigates threat of worms, viruses, DOS attacks, and some intrusions
• Firewall upgrades
– mitigates threats of DOS attacks and some intrusions, worms and viruses
• Redundant HQ Server
– reduces loss of transaction revenue
•Spare laptop computers at each location
– reduces loss of transaction revenue and productivity
• Warranties
– reduces loss of transaction revenue and cost of procuring replacements
• Insurance
– offset cost of liability
• Physical Controls
– reduce probability of theft
• Security Policy
– can be used to reduce most threats.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
43
Case Study
Asset/Vulnerability Matrix
• The coefficients of this matrix are usually based on internal
data as well as financial loss organizations
• For the current example we will assume data for illustration
of the concept
– Transactions are mostly associated with the regional servers which
store the data, the HQ server which takes all requests, and the
network infrastructure with which clients access the data. (.30 each)
– Laptops, desktops and software is only associated with the remaining
10% (.033 each)
– Data that is located on laptops and desktops make up only 10% of
total data because they are only used for collecting and processing.
– The regional servers contain all other data.
– Other assets are associated at 100% with their respective
vulnerabilities. (e.g. laptops with laptops, desktops with desktops,
etc.)
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
44
Case Study
Asset/Vulnerability Matrix, cont’d.
Assets
Transaction
Revenue
Data
(Liability)
Laptops
Desktops
Regional
Servers
HQ Server
Aggregates
(Impact)
10,000
10,000,000
5,000,000
3,000,000
326,000
126,000
 (asset value x
vulnerability)
Laptops
.033
.05
1
0
0
0
5,500,330
Desktops
.033
.05
0
1
0
0
3,500,330
Regional
Servers
.30
.90
0
0
1
0
9,329,000
HQ Servers
.30
0
0
0
0
1
129,000
Network
Infrast.
.30
0
0
0
0
0
3000
Software
.033
0
0
0
0
0
330
Vulnerabilities
Input Asset
Values 
• Customize
matrix to assets & vulnerabilities applicable to case
– Compute cost of each asset and put them in the value row
– Determine correlation with vulnerability and asset
– Compute the sum of product of vulnerability & asset values; add to impact
column
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
45
Case Study
Vulnerability/Threat Matrix
• The coefficients of this matrix are usually based on data
from the literature, e.g.,
–
–
–
–
if rate of failure of hardware is rf (per unit time)
the number of pieces of hardware is n then
the total number of failed components during a time period is rf*n
the fraction of hardware that fails is rf*n/n= rf
• For the current example we will assume data for illustration
of the concept
– Failure rate of laptops is .001 per day (i.e., one in a thousand laptops
encounters hardware failure during a day)
– Similarly failure rate of a desktop is .0002 (i.e. 2 in ten thousand
desktops would encounter hardware failure in a given day.
– Hardware failure can cause loss of software, however, our
assumption is that all software is replaceable from backups
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
46
Case Study
Vulnerability/Threat Matrix, cont’d.
– We assume that the hardware failure will disrupt the network once every
one hundred days
– There is 0.3 percent chance that software failure can lead to failure of
desktops
– We assume that there is a .01 chance of a laptop being stolen, .001 for a
desktop, and .0002 for servers.
– There is a very low chance that network equipment is stolen since it is
kept in secure rooms (.0001)
– When equipment is stolen some software may have been stolen as well
– We assume that denial-of-service is primarily targeted at servers and not
individual machines
– We assume that the denial-of-service can disable machines as well as
cause destruction of software
– Insider attacks are primarily meant to exploit data & disable machines
– We assume that the servers have less access thus are less vulnerable to
insider attacks
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
47
Case Study
Vulnerability/Threat Matrix, cont’d.
Vulnerabilities
Laptops
Desktops
Regional
Servers
HQ
Servers
Network
Infrast.
Software
Aggregates
(Threat
Importance)
5,500,330
3,500,330
9,329,000
129,000
3,000
330
 (impact value x
threat value)
Hardware Failure
.001
.0002
.0002
.0002
.01
0
8,122.00
Software Failure
.003
.003
.003
.003
0
0
55,375.98
Equipment Theft
.0160
.001
.0002
.0002
.0001
.005
93,399.16
Denial of Service
.0001
.0001
.001
.001
0
0
10,358.07
Viruses/Worms
.003
.003
.003
.003
0
.001
55,376.31
Insider Attacks
.001
.001
.0001
.0001
.0001
.001
9,947.09
Intrusion
.001
.001
.001
.001
0
.001
18,458.99
Threats
Input Impact
Aggregates
• Complete matrix based on the specific case
– Add values from the Impact column of the previous matrix
– Determine association between threat and vulnerability
– Compute aggregate exposure values by multiplying impact and the associations
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
48
Case Study
Threat/Control Matrix
• Some of these controls have threats associated with them. However,
these are secondary considerations and we will be focusing on primary
threats.
• We assume that IDS systems will control 30% of the DOS attacks, 30%
of Viruses and Worms and 90% of intrusions
– In addition, IDS systems do not impact insider attacks
• Anti-Virus Software will prevent 90% of Viruses and Worms.
• That upgrades to a firewall will greatly control (90% each) of DOS
attacks, as well as Viruses and Worms. It will control 30% of intrusions,
but not insider attacks.
• A redundant HQ server will control 10% of hardware failure (when the
original HQ server fails). This is the same percentage for theft and
insider attacks.
• Also, a redundant HQ server will help with 80% in cases of DOS attacks
on the HQ server.
• Spare laptops will assist in cases of hardware failure and theft (30%
because of volume).
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
49
Case Study
Threat/Control Matrix, cont’d.
• We assume that warranties will help with 70% of both hardware failure
and software failure. While it will assist with the cost of new hardware or
software, will not reduce employee time.
• It is determined that insurance will be able to control 90% of impacts
from the threats of theft, DOS attacks, Virus/Worm attacks, Insider
Attacks, and Intrusion.
• Physical controls (locks, key cards, biometrics, etc.) will control 90% of
theft.
• Also, it is assumed that a security policy will assist with 20% of all
threats since every policy can have procedures which can assist in
prevention.
• Customize matrix based on the specific case
– Add values from the threat importance column of the previous matrix
– Determine impact of different controls on different threats
– Multiply (1-impact) throughout threat column and multiply to threat
Sanjay Goel, School
of Business/Center
for Information
importance
to get
values.Forensics and Assurance
University at Albany Proprietary Information
50
Case Study
Threat/Control Matrix, cont’d.
Threats
Hardware
Failure
Software
Failure
Theft
Denial of
Service
Viruses/
Worms
Insider
Attacks
Intrusion
Aggregates
Input Threat
Importance Values
8,122.00
55,375.98
93,399.16
10,358.07
55,376.31
9,947.09
18,458.99
 (threat importance
x impact of controls)
Intrusion Detection
0
0
0
.30
.30
0
.90
36,333.41
Anti-Virus
0
0
0
0
.90
0
0
49,838.68
Firewall Upgrades
0
0
0
.90
.90
0
.30
64,698.64
Redundant HQ
Server
.10
0
.10
.80
0
.10
0
19,433.28
Spare Laptops
.30
0
.30
0
0
0
0
30,456.35
Warranties
.70
.70
0
0
0
0
0
44,448.59
Insurance
0
0
.90
.90
.90
.90
.90
168,785.66
Physical Controls
0
0
.90
0
0
0
0
84,059.24
.20
.20
.20
.20
.20
.20
.20
50,207.52
1,228.05
13,290.24
470.73
11.60
31.01
716.19
103.37
Controls
Security Policy
Calculate Exposure
with Controls 
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
51
Case Study
Assignment
•
Given the matrices and the example case provided, use this
same methodology in application to determine the information
security risk in your own organization.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
52
Module 4
Cost Benefit Analysis
& Regression Testing
Cost Benefit Analysis & Regression Testing
Outline
•
•
•
•
How to use matrices for cost benefit analysis?
How to calculate Risk Leverage?
Applying the case study example
Examples
– Unauthorized Access
– Graphical Cost Benefit Analysis with Regression Testing
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
54
Cost Benefit Analysis
Matrix Cost Benefit Analysis
• The exposure before controls is equal to the summation of the
aggregate values for impact value x threat value. (Vulnerability/Threat
Matrix)
– In this case, the value is equal to: $251,037.60
• The exposure after controls is equal to the sum of all of the multiplied
threat importance values.
• For example, in the Hardware Failure column, we will take each of the
threat importance values and subtract them each from 1. These values
should be multiplied together. (Threat/Control Matrix)
– This will give us: (1-.10) x (1 - .30) x (1 - .70) x (1 - .20) = 0.15
– This value will be multiplied by the threat importance value: 0.15 x $8,122.00 =
$1,218.30 (cost with controls of Hardware Failure)
– Do this for all Threat columns and then summate all the values.
– This value is equal to: $15,851.19
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
55
Cost Benefit Analysis
Risk Leverage
• Costs are associated with both:
– Potential Risk Impact
– Reducing Risk Impact
• Risk Leverage is the difference in risk exposure divided by the
cost of reducing the risk
• Let
– rf be the risk exposure after imposing controls
– ri be the risk exposure prior to imposing controls
– c be the cost of controls
Leverage l = (ri-rf)/c
• This tells you how many times the reduction in risk exposure is
greater then the cost of controls.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
56
Cost Benefit Analysis
Matrix Example
• We are using this equation to calculate cost:
–
–
–
–
–
Ci = Csi + Cri x t
Where Ci is the total cost of control i.
Csi is the static (one-time) cost of the control.
Cri is the additional cost per day (maintenance, updates, etc.) for the control.
t is equal to time (if calculating for a year, would equal 365).
• We are assuming cost of control values for this example:
– Intrusion Detection: $21,000 x 11 + $160 x 11 x 365 = $873,400
– Anti-Virus: $1,876 x 4,000 (laptops & desktops) + $1,876 x 11 (number of servers) =
$7,524,636 + 11 x $160 x 365 = $8,167,036
– Firewall Upgrades: $10,000 x 211 + $160 x 211 = $2,143,760
– Redundant HQ Server: $100,000 + $160 x 365 = $158,400
– Spare Laptops: $2,500 x 200 = $500,000
– Warranties (3 year): $100 x 4,000 (laptops & desktops) + $1000 x 10 (regional servers) +
$1,200 (HQ Server) = $411,200
– Insurance: $5,000,000 (per 365 days)
– Physical Controls: $5,000 x 211 + $160 x 211 x 365 = $13,377,400
– Security Policy (creation, implementation, enforcement): $640 x 365 = $233,600
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
57
Cost Benefit Analysis
Matrix Example
• Leverage l = (ri-rf)/c
– ri = $251,037.60 x 365 = $91,628,724
– rf = $15,851.19 x 365 = $5,785,684.35
– C = $30,864,796
• $251,037 – $15,851.19 / $30,864,796 = .008
•
$91,628,724 - $5,785,684.35 / $30,864,796 = 2.78
– The reduction in risk exposure is almost 3x greater than the cost of controls
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
58
Cost Benefit Analysis
Example #4: Unauthorized access
• Scenario: A company uses a common carrier to link
to a network for certain computing applications. The
company has identified the risks of unauthorized
access to data and computing facilities through the
network. These risks can be eliminated by
replacement of remote network access with the
requirement to access the system only from a
machine operated on the company premises. The
machine is not owned; a new one would have to be
acquired.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
59
Cost Benefit Analysis
Example #4: Unauthorized Access
Cost/Benefit Analysis for Replacing Network Access
Item
Risk: unauthorized access and use
Amount
Access to unauthorized data and programs
$100,000 @ 2% likelihood per year
Unauthorized use of computing facilities
$10,000 @ 40% likelihood per year
$2,000
Expected annual loss (2,000 + 4,000)
Effectiveness of network control: 100%
$6,000
-$6,000
$4,000
Cost Benefit Analysis
Example #4: Unauthorized Access
Network Control cost:
Hardware (50,000 amortized over 5 years)
+$10,000
Software (20,000 amortized over 5 years)
+$4,000
Support personnel (each year)
+$40,000
Annual cost
$54,000
Expected annual loss (6,000 – 6,000 +54,000)
$54,000
Savings (6,000 – 54,000)
-$48,000
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
61
Regression Testing
Example #5: Graphical Cost Benefit Analysis
• Scenario: This is a case where use of regression testing is being
considered after making an upgrade to fix a security flaw. We
want to determine if regression testing is economical in this
scenario.
• Regression Testing means applying tests to verify that all
remaining functions are unaffected by the change.
• Lets refer to the diagram on the following slide, to compare the
risk impact of doing regression testing with not doing it.
• Upper part of the diagram
– the risk of conducting regression testing
• Lower part of the diagram
– shows the risks of not doing regression testing
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
62
Regression Testing
Example #5: Cost Savings
• In the two cases, one of three things can
happen if regression is done:
– We find a critical fault
– We miss finding the critical fault
– There are no critical faults to be found.
• For each possibility
– Calculate the probability of an unwanted outcome,
P(UO).
– Associate a loss with that unwanted outcome, L(UO).
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
63
Regression Testing
Example #5: Calculation
In our example, if we do regression testing and miss a critical fault in the system (a
probability of 0.05), the loss could be $30 million. Multiplying the two, we find the
risk exposure for that strategy to be $1.5 million. As the calculations in the figure
prove, it is much safer to do regression testing than to skip it.
Risk Exposure
P(UO) = 0.75
Find critical fault
yes
Do
regression
testing?
no
L(UO) = $0.5M
$0.375M
L(UO) = $30M
P(UO) = 0.05
$1.500M
Don’t find critical fault
L(UO) = $0.5M
P(UO) = 0.20
$0.100M
No critical fault
P(UO) = 0.05
Find critical fault
L(UO) = $0.5M
$0.125M
L(UO) = $30M
P(UO) = 0.75
$16.500M
Don’t find critical fault
L(UO) = $0.5M
P(UO) = 0.20
$0.100M
No critical fault
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
$1.975M
Combined
Risk
Exposure
$16.725M
64
Cost Benefit Analysis and Regression Testing
Questions 1 and 2
1)
What is regression testing?
2)
What is the calculated risk exposure for not doing a regression
testing, if finding a critical fault has a probability of 0.35 and
the loss is estimated at 4.5 million dollars.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
65
Cost Benefit Analysis & Regression Testing
Assignment
•
Do a cost benefit analysis based on the matrix that you have
created for your own organization.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
66
Module 5
Modeling Uncertainties
Modeling Uncertainties
Outline
•
•
How do you model?
Monte Carlo Simulation
–
–
–
–
–
–
–
What is the approach?
How to model valuation of assets?
How to model frequency of threats?
How to model impact of threats?
How to model controls?
How to model distribution of risk exposure?
How to perform a sensitivity analysis for risk exposure?
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
68
Modeling Uncertainties
Modeling Uncertainties
• Uncertainty exists regarding value that should be assumed by
one or more independent variables in the Risk Model.
• Contributions to the model’s uncertainty
– Lack of knowledge about particular values
– Knowledge that some values might always vary
• If it cannot be determined with certainty what value one or
more input variables in a model will assume, this uncertainty
is naturally reflected on the outcome of the dependent
variable(s).
• The risk metric is:
– not determined by the value of its independent variables (asset values
and vulnerabilities, frequency and impact of threats)
– a function of the probability distribution of each of these random
variables
• A good approach to dealing with uncertainty >> simulation
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
69
Modeling Uncertainties
Monte Carlo Simulation: Approach
• The approach follows the following steps:
– Develop risk model
– Define the shape and parameters of probability
distributions of each input variable
– Run Monte Carlo simulation
– Build histogram for dependent variables in the model
(risk and updated risk)
– Compute summary statistics for dependent variables in
model
– Perform sensitivity analysis to detect variability sources
– Analyze potential dependency relationships among
variables in model
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
70
Modeling Uncertainties
Monte Carlo Simulation: Value of Assets
Sample
Mean = 50.00
36.88
44.44
52.00
59.55
67.11
Truncated Normal Distribution(mean = 50)
• Asset values here are samples and do not represent
collected data
– In real cases real assets of the organization need to be
identified
– Value needs to be assigned to the assets
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
71
Modeling Uncertainties
Monte Carlo Simulation: Frequency of Threats
• Annualized frequency of threats is required to compute
the annualized loss expectancy.
• This data can be collected from several sources
– Tracking and collecting data from Internal logs
– Report from agencies such as CERT
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
72
Modeling Uncertainties
Monte Carlo Simulation: Impact of Threats
D3 4
Triangular distribution
(mode, max=1, min=0)
Mean = 0.37
0.00
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
0.25
0.50
0.75
1.00
73
Modeling Uncertainties
Monte Carlo Simulation: Controls
H4 0
Mean = 0.53
0.00
0.25
0.50
0.75
Triangular distribution( mode, max=1, min=0)
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
1.00
74
Modeling Uncertainties
Monte Carlo Simulation: Risk Exposure Distribution
Histogram of Exposure Risk
(1000 runs)
30
25
15
10
5
0
5610
10627
15643
20660
Cumulative Distribution
25677
Risk (in $)
Histogram of Exposure Risk
Cumulative Distribution of Exposure Risk
(1000 runs)
1000
900
800
700
Frequency
Frequency
20
600
500
400
300
200
100
0
5610
10627
15643
20660
25677
Risk (in $)
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
75
Modeling Uncertainties
Monte Carlo Simulation: Reduced Risk Exposure
Histogram of Reduced Exposure Risk
(1000 runs)
45
40
35
25
20
15
10
5
0
47
271
496
720
945
Cumulative Distribution
Risk (in $)
Histogram of Reduced Exposure Risk
Cumulative Distribution of Reduced Exposure Risk
(1000 runs)
1000
900
800
700
Frequency
Frequency
30
600
500
400
300
200
100
0
47
271
496
720
945
Risk (in $)
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
76
Modeling Uncertainties
Monte Carlo Simulation: Sensitivity Analysis
A
n
n
u
a
l
i
z
e
d
F
r
e
q
u
e
n
c
y
Sensitivity Analysis
Exposure Risk
-100.0% -80.0% -60.0% -40.0% -20.0%
0.0%
20.0%
40.0%
60.0%
80.0% 100.0%
Worms
Passw ord Based Attacks
Viruses
Intrusion
Overflow Attacks
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
77
Modeling Uncertainties
Questions 1 and 2
1)
Why does uncertainty exist within risk analysis?
2)
Describe the approach towards Monte Carlo Simulation.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
78
Modeling Uncertainties
Assignment
•
Using the data provided in the case study, or your own risk
analysis, use Monte Carlo Simulation to provide a graphical
display.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
79
Appendix
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
80
Quantitative Analysis
Summary
• Risk Exposure
– RISK EXPOSURE = RISK IMPACT x RISK PROBABILITY
•Annual Loss Expectancy (ALE)
– Identify and determine the value of assets
– Determine vulnerabilities
– Estimate likelihood of exploitation
– Compute ALE
– Survey applicable controls and their costs
– Perform a cost-benefit analysis
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
81
Quantitative Analysis
Summary Cont’d.
•Risk Aggregation:
J
K L
I
i 1
i 1
R*     (t j    ij )  b jk  a kl  al   Ci
j 1k 1l 1
•Optimization
– simple formulation
•Cost Benefit Analysis
I
 ij  (1  qig ij )
k Q
k Q
k 1
j 1
Minimize : R   a j such that  a j  
*
LEVERAGE = (RISK EXPOSUREbefore reduction – RISK EXPOSUREafter reduction)
________________________________________________
COST OF REDUCTION
•Regression Testing
–Used for comparing risk impact
•Monte Carlo Simulation
– 1)Develop risk model, 2) Define the shape and parameters, 3)Run
simulation, 4)Build histogram, 5)Compute summary statistics,
6)Perform sensitivity analysis, 7)Analyze potential dependency
relationship
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
82
Acknowledgements
Grants & Personnel
• Support for this work has been provided through the
following grants
– NSF 0210379
– FIPSE P116B020477
• Damira Pon, from the Center of Information Forensics and
Assurance contributed extensively by reviewing and editing
the material
• Robert Bangert-Drowns from the School of Education
provided extensive review of the material from a pedagogical
view.
Sanjay Goel, School of Business/Center for Information Forensics and Assurance
University at Albany Proprietary Information
83