CostPROF
Cost Risk Analysis
How to adjust your estimate for
historical cost growth
Unit III - Module 9
1
© 2002 SCEA. All rights reserved.
developed by
Unit Index
CostPROF
Unit I – Cost Estimating
Unit II – Cost Analysis Techniques
Unit III – Analytical Methods
6. Basic Data Analysis Principles
7. Learning Curves
8. Regression Analysis
9. Cost Risk Analysis
10. Probability and Statistics
Unit IV – Specialized Costing
Unit V – Management Applications
Unit III - Module 9
2
© 2002 SCEA. All rights reserved.
developed by
Outline
•
•
•
•
•
•
CostPROF
Introduction to Risk
Model Architecture
Historical Data Analysis
Model Example
Summary
Resources
Unit III - Module 9
3
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Introduction to Risk
•
•
•
•
Overview
Definitions
Types of Risk
Risk Process
Unit III - Module 9
4
© 2002 SCEA. All rights reserved.
developed by
Overview
CostPROF
• Risk is a significant part of cost estimation
and is used to allow for cost growth due to
anticipatable and un-anticipatable causes
• There are several approaches to risk
estimation
• Incorrect treatment of risk, while better than
ignoring it, creates a false sense of security
• Risk is perhaps best understood through a
detailed examination of an example method
Unit III - Module 9
5
© 2002 SCEA. All rights reserved.
developed by
Definitions
CostPROF
• Cost Growth:
– Increase in cost of a system from inception
to completion
• Cost Risk:
– Predicted Cost Growth.
In other words:
Cost Growth = actuals
Cost Risk = projections
Unit III - Module 9
6
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Types of Risk
Often implicit or omitted
• Cost Growth = Cost Estimating Growth + Sked/Tech Growth
+ Requirements Growth + Threat Growth
• Cost Risk = Cost Estimating Risk + Sked/Technical Risk +
Requirements Risk + Threat Risk
– Cost Estimating Risk: Risk due to cost estimating errors, and the statistical
uncertainty in the estimate
– Schedule/Technical Risk: Risk due to inability to conquer problems posed
by the intended design in the current CARD or System Specifications
– Requirements Risk: Risk resulting from an as-yet-unseen design shift from
the current CARD or System Specifications arising due to shortfalls in the
documents
• Due to the inability of the intended design to perform the (unchanged) intended
mission
• We didn’t understand the solution
– Threat Risk: Risk due to an unrevealed threat; e.g. shift from the current
STAR or threat assessment
1
• The problem changed
2
Unit III - Module 9
7
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Basic Flow of the Risk Process
Inputs
Structure & Execution
Includes the organization,
the mathematical assumptions,
and how the model runs
From the cost analyst
and technical experts
• The CARD
• Expert rating/scoring
• Point Estimate
Outputs
To the decision maker
and the cost analyst
• Means
• Standard Deviations
• Risk by CWBS
Inputs and outputs, although outside the
purview of the risk analyst, are determined by
the structure and execution of the risk model
Unit III - Module 9
8
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Engineers’ and Cost Analysts’ View of Risk
•
Engineers
Work in physical materials,
with
–
–
•
•
Physics-based responses
Physical connections
–
–
Typically examine or discuss a
specific outcome
–
–
–
•
System Parameters
Designs
•
Given this solution, what will go
wrong?
Are my design margins
enough?
Statistical relationships
Correlation
Typically examine or discuss a
general outcome set
–
–
Typically seek to know:
–
•
Cost Analysts
Work in dollars and parameters,
with
Probability distribution
Statistical parameters such as
mean and standard deviation
Typically seek to know:
– Given this relationship, what is the
range of possibilities?
– Are my cost margins enough?
Unit III - Module 9
9
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Model Architecture
• Inputs
• Structure
• Execution
Unit III - Module 9
10
© 2002 SCEA. All rights reserved.
developed by
CostPROF
–
Computation
–
Coverage & Partition
•
•
•
•
Cost Estimating
Schedule / Technical
Requirements
Threat
Inputs
Structure
Assigning Cost to Risk
•
•
CERs
Direct Assessment of
Distribution Parameters
• Factors
• Rates
Below-the-Line
•
•
–
Probability Model
Organization
–
Interval w/ objective criteria
Interval
Ordinal
None
–
Yes
No
• Monte Carlo
• Method of Moments
• Deterministic
Execution
• Historical
• Domain Experts
• Conceptual
Distribution
•
•
•
•
•
Cross
Checks
Scoring
•
•
•
•
Dollar
Basis
General Model Architecture
Normal
Log Normal
Triangular
Beta
Bernoulli
Correlation
•
•
•
•
Functional
Relational
Injected
None
• Means
• CVs
• Inputs
Tip: Higher is
better except in
Cross Checks
Unit III - Module 9
11
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Inputs – Scoring
• Interval with objective criteria
8
– Set scoring based on objective criteria, and for which the
distance (interval) between scores has meaning. (Note:
the below example is also Ratio, because it passes
through the origin.)
• A schedule slip of 1 week gets a score of 1, a slip of 2 weeks gets
a score of 2, a slip of 4 weeks gets a 4, a slip of 5 weeks gets a
score of 5, etc.
• The difference between a score of 1 and 2 is as big as a
difference between score of 4 and 5
• A scale is interval if it acts interval under
examination*
SCORING
“Nominal, ordinal, interval, and ratio typologies are misleading,” P.F. Velleman
and L. Wilkinson, The American Statistician, 1993, 47(1), 65-72
Unit III - Module 9
© 2002 SCEA. All rights reserved.
INPUTS
ORGANIZATION
PROB.
MODEL
COMPUTATION
12
CROSS
CHECKS
developed by
Inputs – Scoring
CostPROF
• Interval
– Set scoring for which the distance (interval)
between scores has meaning
• Low risk is assigned a 1, medium risk is assigned a 5,
and a high risk is assigned a 10
• Note that it is not immediately clear that the scale is
interval, but it is surely not subject to objective criteria.
• Ordinal
– Score is relative to the measurement
• e.g., difficulty in achieving schedule is high, medium, or
low
• None
Unit III - Module 9
13
© 2002 SCEA. All rights reserved.
developed by
Inputs – Dollar Basis
•
CostPROF
Historical
– Actual costs of similar programs or components of
programs are used to predict costs
•
Domain Experts
– Persons with expertise regarding similar programs
or program components assess the cost based on
their experience
•
Conceptual
– An arbitrary impact is assigned
• Any scale without a historical basis or expert assessment
is conceptual
SCORING
INPUTS
Unit III - Module 9
© 2002 SCEA. All rights reserved.
ORGANIZATION
PROB.
MODEL
COMPUTATION
14
CROSS
CHECKS
developed by
CostPROF
Org – Coverage & Partition
• How the four types of risk are covered
and partitioned
–
–
–
–
Cost Estimating
Schedule/Technical
Requirements
Threat
These risk types may be covered implicitly or
explicitly in any combination.
Unit III - Module 9
© 2002 SCEA. All rights reserved.
SCORING
INPUTS
ORGANIZATION
PROB.
MODEL
COMPUTATION
15
CROSS
CHECKS
developed by
CostPROF
Org – Assigning Cost to Risk
• Risk CERs: Equations are developed that reflect the
relationship between an interval risk score and the
cost impact of the risk (this might also be termed a
Risk Estimating Relationship (RER))
9
– These equations amount to the same thing as CERs used in
the cost estimate
– e.g., Risk Amount = 0.12 * Risk Score
• Direct Assessment of Distribution Parameters: Costs
are captured in shifts of parameters of the risk, e.g.,
shifted end points for triangulars, shifted end points or
means for betas, etc.
– Note: Scoring is completely eliminated from this mapping
method
SCORING
INPUTS
– e.g., triangles assessed by domain experts
Unit III - Module 9
© 2002 SCEA. All rights reserved.
ORGANIZATION
PROB.
MODEL
COMPUTATION
16
CROSS
CHECKS
developed by
CostPROF
Org – Assigning Cost to Risk
• Factors: Fractions or percents are used in
conjunction with the scores and the cost of the
component or program
–
–
–
–
e.g., a score of 2 increases the cost of the component by 8%
Antenna Risk Score = 2
Cost of Antenna = $4090K
Risk Amount = 0.08 * 4090K = $327.2K
• Rates: Predetermined costs are
associated with the scores
–
–
–
–
e.g., a score of 2 has a cost of $100K
Antenna Risk Score = 2
Cost of Antenna = $4090K
Risk Amount = $100K
Unit III - Module 9
© 2002 SCEA. All rights reserved.
Warning: Rates are
independent of the
element’s cost.
SCORING
INPUTS
ORGANIZATION
PROB.
MODEL
COMPUTATION
17
CROSS
CHECKS
developed by
CostPROF
Org – Below-the-Line
• Below-the-Line Elements
– Elements that are driven by hardware, software,
and the like
– Below-the-Line Elements include:
• Systems Engineering/Program Management (SE/PM)
• System Test and Evaluation (ST&E)
– Not all models account for this cost growth
– Functional Correlation is another approach to
address the risk in these elements
9
Unit III - Module 9
© 2002 SCEA. All rights reserved.
SCORING
INPUTS
ORGANIZATION
PROB.
MODEL
COMPUTATION
18
CROSS
CHECKS
developed by
CostPROF
Probability Model – Distribution
• Normal
10
4
• Lognormal
– Best behavior, most
iconic
– Theoretically (although
not practically) allows
negative costs, which
spook some users
– Symmetric, needs mean
shift to reflect propensity
for positive growth
– A natural result in
non-linear CERs
– Indistinguishable
from Normal at CVs
below 25%
– Skewed
Unit III - Module 9
© 2002 SCEA. All rights reserved.
SCORING
INPUTS
ORGANIZATION
PROB.
MODEL
COMPUTATION
19
CROSS
CHECKS
developed by
CostPROF
Probability Model – Distribution
• Triangular
–
–
10 –
–
Most common
Easy to use, easy to understand
Modes, medians do not add
Skewed
• Beta
– Rare now, but formerly popular
– Solves negative cost and duration issues
– Many parameters – simplifications like PERT
Beta are possible
– Skewed
• Bernoulli
–
Probability is only assigned to two possible
outcomes, success and failure (p and 1-p)
– Simplest of all discrete distributions
– Mean = p
– Variance = p*(1-p)
Unit III - Module 9
20
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Probability Model – Correlation
Correlation is a measure of the relation between
two or more variables/WBS elements
• Functional: Arises between source and derivative
variables as a result of functional dependency. The
lines of the Monte Carlo are cell-referenced wherever
relationships are known.
– CERs are entered as equations
3
– Cell references are left in the spreadsheet
– When the Monte Carlo runs, input variables
fluctuate, and outputs of CERs reflect this
An Overview of Correlation and Functional Dependencies in Cost Risk and
Uncertainty Analysis, R. L. Coleman and S. S. Gupta, DoDCAS, 1994
Unit III - Module 9
© 2002 SCEA. All rights reserved.
SCORING
INPUTS
ORGANIZATION
PROB.
MODEL
COMPUTATION
21
CROSS
CHECKS
developed by
CostPROF
Functional Correlation
• New: Simulation run with
functional dependencies
entered as they are in
cost model
400
400
350
350
SEPM
SEPM
• Old: No Functional
Correlation; Simulation
run with WBS items
entered as values
300
250
200
1000
300
250
1200
1400
1600
1800
200
1000
2000
1200
1400
1600
1800
Recurring Production
Recurring Production
Not Correlated
Correlated
2000
Note shift of mean, and increased variability
Unit III - Module 9
22
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Probability Model – Correlation
• Relational: Introduces the geometry of
correlation and provides a substantial
improvement over injected correlations, and
fills a gap in FC
– Relational Correlation provides insight into
• the tilt of the data, i.e. the regression line,
• and the variance around the regression line
Relational Correlation: What to do when Functional Correlation is Impossible, R.
L. Coleman, J. R. Summerville, M. E. Dameron, C. L. Pullen, S. S. Gupta,
ISPA/SCEA Joint International Conference,2001
Unit III - Module 9
23
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Probability Model - Correlation
• Injected: Imposed by setting the
correlation directly between variables
without having a functional relationship.
• None: No relationship exists among the
variables. The lines of the Monte Carlo
are self contained.
Unit III - Module 9
24
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Shortcomings of Injected Correlation
• Correlations are very hard to estimate
• No check of the functional implications of the
correlations is done
– This is troublesome because of the regression line
that arises when we insert a correlation.
– Simply injecting arbitrary correlations of 0.2 - 0.3
to achieve dispersion is unsatisfactory as well.
• Unless the injected correlations are among elements that
are actually correlated
• If correlations are actually known, no harm is
done.
Unit III - Module 9
25
© 2002 SCEA. All rights reserved.
developed by
Execution – Computation
CostPROF
• Monte Carlo: A widely accepted method, used on a broad
range of risk assessments for many years. It produces cost
distributions. The cost distributions give decision makers insight
into the range of possible costs and their associated probabilities.
• Method of Moments: The mean and standard
deviation of lower-level WBS lines are known, and are rolled up
assuming independence to provide higher-level distributions.
10
–
–
–
–
Only provides an analysis of distribution at a top level
Easy to calculate
Negated by the rapid advances in microcomputer technology
Only works for independent elements, unless covariances are allowed
for, which is difficult.
• Deterministic: Only point values are used.
other probabilistic effects are taken into account.
Unit III - Module 9
© 2002 SCEA. All rights reserved.
No shifts or
SCORING
INPUTS
ORGANIZATION
PROB.
MODEL
COMPUTATION
CROSS
CHECKS
developed
by
26
CostPROF
Risk Assessment Techniques
• Add a Risk Factor/Percentage (Minutes)
– Low accuracy, no intervals
• Bottom Line Monte Carlo/Bottom Line Range/Method of Moments
(Hours)
– Moderate accuracy, provides intervals
• Historically based Detailed Monte Carlo (Months of non-recurring
work, but recurring in days)
– Time consuming non-recurring work, but with recurring implementation
being easier, accurate if done right. Provides intervals.
• Expert Opinion-Based Probability and Consequence (Pf*Cf) or
Expert Opinion-Based Detailed Monte Carlo (Months)
– Time consuming with no gains in recurring effort, but accurate if done
right. Provides intervals.
• Detailed Network and Risk Assessment (Month)
– Time consuming with no gains in recurring effort, but accurate if done
right. Provides intervals.
Unit III - Module 9
27
© 2002 SCEA. All rights reserved.
developed by
Execution – Cross Checks
CostPROF
• Means: The mean cost growth factor for WBS items
can be compared to history as a way to cross check
results
• CVs: The CV of the cost growth factors for WBS items
can be compared to history as a way to cross check
results
• Inputs: Checks are performed on inputs or other
parameters to see if historical values are in line with
11
program assumptions
– Example: Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic, and to
see if the underlying database is
SCORING
INPUTS
representative of the program.
Unit III - Module 9
© 2002 SCEA. All rights reserved.
ORGANIZATION
PROB.
MODEL
COMPUTATION
28
CROSS
CHECKS
developed by
CostPROF
Historical Data Analysis
• SARs
• Contract Data
• Common Problems
Unit III - Module 9
29
© 2002 SCEA. All rights reserved.
developed by
Intro to SARs – Sample
Sample Program: XXX, December 31, 19XX
12
CostPROF
A SAR report is
submitted for each
year of a program’s
Acquisition cycle.
The most recent
SAR is used to
determine cost
growth
To calculate the CGF,
adjust the current
estimate for quantity
changes, then divide
by the baseline
estimate
Unit III - Module 9
30
© 2002 SCEA. All rights reserved.
developed by
Contract Data
CostPROF
• Hard to use – problems with changing baselines, lack of
reasons for variances, and access to data
• Preliminary comparative analysis suggests Contract
Data mimics patterns in SAR data
– Shape of distribution
– Trends in tolerance for cost growth
• K-S tests find no statistically significant difference
between Contract data and SAR data for programs
<$1B in RDT&E
– Failed to reject the null hypothesis of identical distributions
• Descriptive statistics indicate amount of Contract Data
growth and dispersion is more extreme than previously
found in SAR studies
• SAR data remains the best choice for analysis and
predictive modeling
NAVAIR Cost Growth Study: A Cohorted Study of The Effects of Era,
Size, Acquisition Phase, Phase Correlation and Cost Drivers , R. L.
Coleman, J. R. Summerville, M. E. Dameron, C. L. Pullen, D. M. Snead,
Unitand
III ISPA/SCEA
- Module 9 International Conference, 2001
DoDCAS, 2001
© 2002 SCEA. All rights reserved.
31
developed by
CostPROF
Contract Data Exploratory Analysis
50.0
40.0
30.0
Contract
Data
Contract
20.0
10.0
10.0
CGF vs init est -- NAVAIR Contract AND RAND 93
CGF
vs IPE-Contract and SAR (RDT&E)
(RDT&E)
ZOOM
IN
with
common
Scale
ZOOM IN w/COMMON
SCALE
8.0
0.0
$0
$100
$200
$300
$400
$500
$600
8.0
6.0
4.0
7.0
6.0
2.0
5.0
RAND93
SAR
Data
4.0
0.0 $0
3.0
$200
$400
$600
Contract Data
RAND93
$800
$1,000
2.0
1.0
0.0
$0
$1,000
$2,000
$3,000
$4,000
$5,000
$6,000
$7,000
Contract Data blends well: Continues trend that tolerance for growth
increases as program size decreases
Unit III - Module 9
32
© 2002 SCEA. All rights reserved.
developed by
Common Problems
CostPROF
• Most historically-based methods rely on
SARs
– Adjusting for quantity – important to remove
quantity changes from cost growth
– Beginning points – the richest data source is found
by beginning with EMD
– Cohorting must be introduced to avoid distortions
15 • EVM data is also potentially useable, but re-
baselined programs are a severe
complication.
• “Applicability” and “currency” are the most
common criticisms
Unit III - Module 9
33
© 2002 SCEA. All rights reserved.
developed by
Applicability and Currency
CostPROF
• Applicability: “Why did you include that
in your database?”
– Virtually all studies of risk have failed to
find a difference among platforms (some
exceptions)
– If there is no discoverable platform effect,
more data is better
• Currency: “But your data is so old!”
– Previous studies have found that post-1986
data is preferable
– Data accumulation is expensive
Unit III - Module 9
34
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Model Example
•
•
•
•
Overview
Scoring
Database
Sample Outputs
Unit III - Module 9
35
© 2002 SCEA. All rights reserved.
developed by
CostPROF
–
–
Computation
13
Interval
Ordinal
None
Coverage & Partition
•
•
•
•
Cost Estimating
Schedule / Technical
Inputs
Structure
Requirements
Threat
Assigning Cost to Risk
•
•
CERs
Direct Assessment of
Distribution Parameters
• Factors
• Rates
Below-the-Line
•
•
–
Probability Model
Organization
–
Interval w/ objective criteria
–
Yes
No
• Monte Carlo
• Method of Moments
• Deterministic
Execution
• Historical
• Domain Experts
• Conceptual
Distribution
•
•
•
•
•
Cross
Checks
Scoring
•
•
•
•
Dollar
Basis
Example Model Architecture
Normal
Log Normal
Triangular
Beta
Bernoulli
Correlation
•
•
•
•
Functional
Relational
Injected
None
• Means
• CVs
• Inputs
Tip: Higher is
better except in
cross checks
Unit III - Module 9
36
© 2002 SCEA. All rights reserved.
developed by
Assessment Approach
CostPROF
• Develop a cost estimating risk distribution for
each CWBS element
• Develop a schedule/technical risk distribution for
each WBS entry for:
– Hardware
– Software
– Note that “Below-the-line” WBS elements get risk
from Above-the-line WBS elements via Functional
Correlation
• Combine these risk distributions and the point
estimate using a Monte Carlo simulation
Unit III - Module 9
37
© 2002 SCEA. All rights reserved.
developed by
Example Model in Blocks
CostPROF
Cost Estimating Risk
IPE
CARD
Risk
Scoring
Standard
Errors &
SEEs
Functional
Correlation
Mapping
Monte
Carlo
Sked/Tech Risk
Risk
Report
Cost Risk Analysis of the Ballistic Missile Defense (BMD) System, An Overview of New
Initiatives Included in the BMDO Risk Methodology, R. L. Coleman, J. R. Summerville,
D. M. Snead,
S. III
S. -Gupta,
G. E.
38
Unit
Module
9 Hartigan, N. L. St. Louis, DoDCAS, 1998 (Outstanding
Contributed Paper), and ISPA/SCEA International Conference, 1998
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Cost Estimating Risk Assessment
• Consists of a standard deviation and a
bias associated with the costing
methodologies
14
– Standard deviation comes from the CERs
and factors
– Bias is a correction for underestimating
Unit III - Module 9
39
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Sked/Tech Risk Assessment
• Technical risk is decomposed into
categories and each category into sub
categories
– Hardware sub categories:
• Technology Advancement, Engineering
Development, Reliability, Producibility,
Alternative Item and Schedule
– Software sub categories:
• Technology Approach, Design Engineering,
Coding, Integrated Software, Testing,
Alternatives, and Schedule
Unit III - Module 9
40
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Hardware Risk Scoring Matrix
Risk
Categories
1
2
Technology
Advancement
Completed (State
of the Art)
Engineering
Development
Completed
(Fully Tested)
Reliability
Historically High
for Same Item
Historically High
on Similar Items
Producibility
Production &
Yield Shown on
Same Item
Alternate
Item
Exists or
Availability on
Other Items Not
Important
Production &
Yield Shown on
Similar Items
Exists or
Availability of
Other Items
Somewhat
Important
Schedule
Easily Achievable
3
4
5
6
0
Risk Scores (0=Low, 5=Medium, 10=High)
1-2
3-5
6-8
Minimum
Modest
Significant
Advancement
Advancement
Advancement
Required
Required
Required
HW/SW
Prototype
Detailed Design
Development
Achievable
9-10
New Technology
Concept Defined
Known Modest
Problems
Known Serious
Problems
Unknown
Production &
Yield Feasible
Production
Feasible & Yield
Problems
No Known
Production
Experience
Potential
Alternative Under
Development
Potential
Alternative in
Design
Alternative Does
Not Exist & is
Required
Somewhat
Challenging
Challenging
Very Challenging
Unit III - Module 9
41
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Calculating Sked/Tech Risk Endpoints
• Technical experts score each of the
categories from 0 (no risk) to 10 (high risk)
• Each category is weighted depending on the
relevancy of the category
– Weights are allowed, but rarely used
• Weighted average risk scores are mapped to
a cost growth distribution
– This distribution is based on a database of cost
growth factors of major weapon systems collected
from SARs. These programs range from those
which experienced tremendous cost growth due to
technical problems to those which were well
managed and under budget.
Unit III - Module 9
42
© 2002 SCEA. All rights reserved.
developed by
Sked/Tech Score Mapping
CostPROF
Distribution End Points
Typical Risk Assessment Score Mapped to Factor-RDT&E
3.5
3
2.5
2
1.5
1
0.5
0
0
1
2
3
5
6
7
8
9
10
Risk Score
AVG=1.17
MIN=0.77
4
AVG=1.28
MAX=1.58
AVG=1.40
MIN=0.61
MAX=1.96
MIN=0.46
MAX=2.34
Unit III - Module 9
43
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Sked/Tech Risk Distribution
MIN=0.46
MIN=0.61
MIN=0.77
MAX=1.96
MAX=1.58
MAX=2.34
These are the PDFs for 3 risk
scores above. More risk has
higher mode, wider base, all are
symmetric.
This is the composite
PDF for all SARs
Bars are the frequency
of occurrence of each
risk score
Model
1
AVG=1.40
AVG=1.28
AVG=1.17
3
5
7
9
10
S/T Risk Score
Unit III - Module 9
44
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Cost Growth Database
28
Risk appears skewed,
perhaps Triangular or
Lognormal
20
15
8
9
7
4
5
4
3
2
-30%
1
-20%
-10%
Cost Decrease
0%
10%
20%
30%
40%
50%
60%
1
70%
80%
90%
100% 400%
Cost Increase
No Change
This distribution, found in databases, is the
result of a blending of a family of
distributions as shown on the previous
45
Unit III - Module 9
slide.
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Risk Report Sample Output
Crystal Ball Report
Simulation started on 1/3/01 at 18:24:18
Simulation stopped on 1/3/01 at 18:24:47
Forecast: DD 21$
5
Statistics:
Trials
Mean
Median
Mode
Standard Deviation
Variance
Skewness
Kurtosis
Coeff. of Variability
Range Minimum
Range Maximum
Range Width
Mean Std. Error
Value
5000
172.12
172.33
--43.32
1877.03
0.01
2.88
0.25
22.38
318.94
296.56
0.61
Unit III - Module 9
46
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Example Cost Estimate with Risk – R&D
Note: These are means – there is an
associated confidence interval, not
portrayed.
33.7%
150
25%
8.7%
$
S/T Risk
100
CE Risk
50
6
7
Init Pt Est
0
Initial Point
Estimate
Add Cost
Estimating
Risk
Add
Sched/Tech
Risk
Unit III - Module 9
47
© 2002 SCEA. All rights reserved.
developed by
Summary
CostPROF
– Why include risk?
• Risk adjusts the cost estimate so that it more closely
represents what historical data and experts know to be
true … it predicts cost growth
– How to treat risk?
• We have seen an overview of the many different options
in terms of inputs, the structure of the risk model, and
how to execute the risk model
• The choices are varied, but it is important that the model
“fits together” and that it predicts well.
– Closing thought: Always include cross checks to
support the accuracy of the model and the specific
results for a program
• The model may seem right, but will it (did it) predict
accurate results?
Unit III - Module 9
48
© 2002 SCEA. All rights reserved.
developed by
Risk Resources – Books
CostPROF
•Against the Gods: The Remarkable Story of Risk, Peter
L. Bernstein, August 31, 1998, John Wiley & Sons
•Living Dangerously! Navigating the Risks of Everyday
Life, John F. Ross, 1999, Perseus Publishing
•Probability Methods for Cost Uncertainty Analysis: A
Systems Engineering Perspective, Paul Garvey, 2000,
Marcel Dekker
•Introduction to Simulation and Risk Analysis, James R.
Evan, David Louis Olson, James R. Evans, 1998,
Prentice Hall
•Risk Analysis: A Quantitative Guide, David Vose, 2000,
John Wiley & Sons
Unit III - Module 9
49
© 2002 SCEA. All rights reserved.
developed by
Risk Resources – Web
CostPROF
• Decisioneering
– Makers of Crystal Ball for Monte Carlo
simulation
– http://www.decisioneering.com
• Palisade
– Makers of @Risk for Monte Carlo
simulation
– http://www.palisade.com
Unit III - Module 9
50
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Risk Resources – Papers
•Approximating the Probability Distribution of Total System Cost,
Paul Garvey, DoDCAS 1999
• Why Cost Analysts should use Pearson Correlation, rather than
Rank Correlation, Paul Garvey, DoDCAS 1999
• Why Correlation Matters in Cost Estimating , Stephen Book,
DoDCAS 1999
•General-Error Regression in Deriving Cost-Estimating
Relationships, Stephen A. Book and Mr. Philip H. Young, DoDCAS
1998
• Specifying Probability Distributions From Partial Information on
their Ranges of Values, Paul R. Garvey, DoDCAS 1998
•Don't Sum EVM WBS Element Estimates at Completion, Stephen
Book, Joint ISPA/SCEA 2001
•Only Numbers in the Interval –1.0000 to +0.9314… Can Be Values
of the Correlation Between Oppositely-Skewed Right-Triangular
Distributions, Stephen Book , Joint ISPA/SCEA 1999
Unit III - Module 9
51
© 2002 SCEA. All rights reserved.
developed by
Risk Resources – Papers
CostPROF
•An Overview of Correlation and Functional Dependencies in Cost
Risk and Uncertainty Analysis, R. L. Coleman, S. S. Gupta,
DoDCAS, 1994
•Weapon System Cost Growth As a Function of Maturity, K. J.
Allison, R. L. Coleman, DoDCAS 1996
•Cost Risk Estimates Incorporating Functional Correlation,
Acquisition Phase Relationships, and Realized Risk, R. L.
Coleman, S. S. Gupta, J. R. Summerville, G. E. Hartigan, SCEA
National Conference, 1997
•Cost Risk Analysis of the Ballistic Missile Defense (BMD) System,
An Overview of New Initiatives Included in the BMDO Risk
Methodology, R. L. Coleman, J. R. Summerville, D. M. Snead, S. S.
Gupta, G. E. Hartigan, N. L. St. Louis, DoDCAS, 1998 (Outstanding
Contributed Paper) and ISPA/SCEA International Conference,
1998
Unit III - Module 9
52
© 2002 SCEA. All rights reserved.
developed by
Risk Resources – Papers
CostPROF
•Risk Analysis of a Major Government Information Production
System, Expert-Opinion-Based Software Cost Risk Analysis
Methodology, N. L. St. Louis, F. K. Blackburn, R. L. Coleman,
DoDCAS, 1998 (Outstanding Contributed Paper), and ISPA/SCEA
International Conference, 1998 (Overall Best Paper Award)
•Analysis and Implementation of Cost Estimating Risk in the
Ballistic Missile Defense Organization (BMDO) Risk Model, A Study
of Distribution, J. R. Summerville, H. F. Chelson, R. L. Coleman, D.
M. Snead, Joint ISPA/SCEA International Conference 1999
•Risk in Cost Estimating - General Introduction & The BMDO
Approach, R. L. Coleman, J. R. Summerville, M. DuBois, B. Myers,
DoDCAS, 2000
•Cost Risk in Operations and Support Estimates, J. R. Summerville,
R. L. Coleman, M. E. Dameron, SCEA National Conference, 2000
Unit III - Module 9
53
© 2002 SCEA. All rights reserved.
developed by
Risk Resources – Papers
CostPROF
•Cost Risk in a System of Systems, R.L. Coleman, J.R. Summerville, V.
Reisenleiter, D. M. Snead, M. E. Dameron, J. A. Mentecki, L. M. Naef,
SCEA National Conference, 2000
• NAVAIR Cost Growth Study: A Cohorted Study of the Effects of Era, Size,
Acquisition Phase, Phase Correlation and Cost Drivers, R. L. Coleman, J.
R. Summerville, M. E. Dameron, C. L. Pullen, D. M. Snead, ISPA/SCEA
Joint International Conference, 2001
•Probability Distributions of Work Breakdown Structures,, R. L. Coleman, J.
R. Summerville, M. E. Dameron, N. L. St. Louis, ISPA/SCEA Joint
International Conference, 2001
•Relational Correlation: What to do when Functional Correlation is
Impossible, R. L. Coleman, J. R. Summerville, M. E. Dameron, C. L. Pullen,
S. S. Gupta, ISPA/SCEA Joint International Conference,2001
•The Relationship Between Cost Growth and Schedule Growth, R. L.
Coleman, J. R. Summerville, DoDCAS, 2002
•The Manual for Intelligence Community CAIG Independent Cost Risk
Estimates, R. L. Coleman, J. R. Summerville, S. S. Gupta, DoDCAS, 2002
Unit III - Module 9
54
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Advanced Topics
• Relational Correlation and the
Geometry of Regression
Unit III - Module 9
55
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Geometry of Bivariate Normal Random Variables
The dispersion and axis tilt of the “data cloud”
is a function of correlation:
• less correlation, more dispersion about the
axis
• more correlation, more axis tilt
y
ρ=.75
σy
(μx, μy)
μy
ρ=0
σy
σx
σx
μx
x
Unit III - Module 9
56
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Implications for Regression Line
y
This line is with perfect correlation …
The slope that would be true if ρ = 1
y = ρ(σy / σx) (x- μx) + μy
ρ=.75
2σx
σy
(μx, μy)
μy
2σy
σy
b
This line has
correlation added
b= μy- ρσy / σx * μx
σx
σx
μx
x
Unit III - Module 9
57
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Geometry of Regression Line
y
Slope m varies with
ρ, σx, σy
The regression line of y on x
depends on their means, their
standard deviations and their
correlation
y = ρ(σy / σx) (x- μx) + μy
ρ=1
σy
(μx, μy)
μy
2σy
σy
b
b= μy- ρσy / σx * μx
ρ=0
Dispersion varies
with ρ
σx
Intercept b varies with
ρ, σx, σy, μx, and μy
Range of slopes
Range of intercepts
2σx
σx
μx
ρ=-1
x
Unit III - Module 9
58
© 2002 SCEA. All rights reserved.
developed by
CostPROF
Geometry of r squared
r2 is the percent reduction
between these two variances:
σy2 and σy|x2
or
σx2 and σx|y2
y
μy
σy
σy|x
σy
σy|x
b
r2 = 0.75
σy|x
σy|x
r2 = 0
Variance of y|x = (1- ρ2)* σy2
σx
σx
Unitμx
III - Module 9
© 2002 SCEA. All rights reserved.
x
59
developed by