Overview of Addressing Risk with
COSYSMO
Garry Roedler & John Gaffney
Lockheed Martin
March 17, 2008
Agenda
• This presentation describes:
– Risk Enhancements made by Lockheed
Martin to the “Academic COSYSMO” systems
engineering estimation model/tool.
– The results from the COSYSMO Workshop at
the PSM meeting in July 2007.
2
The Situation
• Uncertainty is a fact of business life.
• Key business and technical decisions need to be
made in the face of uncertainty, e.g., cost and
schedule.
• Yet, often, business capture and execution
teams develop only “the” point estimates for
effort and schedule without any statement of:
– The degree of uncertainty in these values
– The overrun exposure they imply.
• This undoubtedly contributes to the number
of red programs
3
Background
• Risk and reuse enhancements for SE cost
estimation have been Implemented in the LM
version of the COSYSMO tool, “COSYSMOR,”
or “COSYSMO Risk and Reuse.”
– The enhancements relate to concerns that have been
discussed at various COSYSMO working group and
PSM meetings.
• Major driver was to get away from “single point”
cost estimates
– Better recognize the uncertainty associated with
estimates.
4
Summary of Enhanced COSYSMOR Functions
The COSYSMOR model/tool provides four major functions
in addition to those provided by Academic COSYSMO:
1.
2.
3.
4.
Estimation of Cost/Effort and Schedule Uncertainties: “Risk”
and “Confidence”: Provides quantification of the impacts of
uncertainties in the values of key model parameter values. They
are multiple cost and schedule values with associated
probabilities.
Representation of Multiple Types of Size Drivers: Provides for
entering counts of: new, modified, adopted (reused), and deleted
types for each of the four size driver categories.
Labor Scheduling: Provides the spread of systems engineering
labor for five systems engineering activities and across four
development phases (time).
Labor Allocation: Provides for the user to select the percentage
allocations of the twenty activity/phase pairs or effort elements.
This presentation will focus on just the Risk function.
5
The COSYSMOR User Enters 3-Value Estimates For:
•
•
Model Parameters A and E
Scope or Project Size Characteristics, Equivalent Size Drivers:
–
–
–
–
•
Number of System Requirements
Number of System Interfaces
Number of System-Specific Algorithms
Number of Operational Scenarios
Cost/Performance Characteristics, Cost Drivers:
–
–
–
–
–
–
–
–
–
–
–
–
–
–
Requirements Understanding
Architecture Understanding
Level of Service Requirements
Migration Complexity
Technology Risk
Documentation
# and diversity of installations/platforms
# of recursive levels in the design
Stakeholder team cohesion
Personnel/team capability
Personnel experience/continuity
Process capability
Multi-site coordination
Tool Support
6
COSYSMOR Data Entry
ENTER SIZE PARAMETERS FOR SYSTEM OF INTEREST
Low
Likely*
Easy
# of System Requirements
# of System Interfaces
# of Algorithms
# of Operational Scenarios
Equivalent Total Requirements Size
Equivalent New Requirements Size
Requirements Understanding
Architecture Understanding
Level of Service Requirements
Migration Complexity
Technology Risk
Documentation
# and diversity of installations/platforms
# of recursive levels in the design
Stakeholder team cohesion
Personnel/team capability
Personnel experience/continuity
Process capability
Multisite coordination
Tool support
Composite effort multipliers
Nominal
Difficult
Easy
Nominal
Difficult
10
1
10
11
2
10
3
2
11
4
9
2
4
10
3
5
4
2
5
5
351.90
402.90
299.58
341.39
SELECT COST PARAMETERS FOR SYSTEM OF INTEREST
9
2
3
4
VH
H
L
N
L
L
N
VL
H
H
VH
VH
VH
H
Low Value
0.60
0.81
0.79
1.00
0.84
0.91
1.00
0.80
0.81
0.81
0.67
0.77
0.80
0.85
0.05
N
N
N
N
N
N
N
N
N
N
N
N
N
N
Likely Value*
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
High
Easy
Nominal
11
2
5
4
High Value
1.36
L
1.27
L
1.00
N
1.92
EH
1.32
H
1.28
VH
1.86
EH
1.21
H
1.50
VL
1.48
VL
1.21
L
1.21
L
1.15
L
1.16
L
54.58
Three-point data entry for size and cost drivers.
Difficult
12
13
11
6
1
5
7
6
520.40
437.77
Size Para
* Note: If you do not wish to use th
for the size and cost parameters, b
then enter such values in the "Like
simply ignore the range estimates
equal to the Likely value.
** You can specify the proportions
their cost per requirement relative
you can enter your estimate in the
category, e.g., "Systems Requirem
these three data entries automatic
of each requirements category. Th
the course of execution of a projec
EAC (Estimate-At-Completion).The
figure were 10%, that would mean
7
COSYSMOR Data Entry (Cont’d)
COSYSMO MODEL PARAMETERS
Equivalent Size, S (=Equivalent New)
Unit Effort Constant, A:Baseline
Unit Effort Constant, A:User**
Unit Effort Constant, A Selected
Size Exponent, E
Cost Parameter Product,D
SYSTEMS ENGINEERING PERSON MONTHS
SYSTEMS ENGINEERING PERSON HOURS
Low
318
38.550
49.136
49.136
1.000
0.364
37.4
5679
Likely
338
38.550
49.136
49.136
1.060
1.202
186.6
28361
High
Nominal *
396
341
38.550
38.55
49.136
38.55
49.136
38.55
1.100
1.06
3.752
1.00
874.7
122.9
132952 18675
COSYSMO MODEL
FORM
PM, PH=A*(SE)*D
PM=Person Months
PH=Person Hours
Three-point data entry effort (A) and exponent (E) values
8
Person Hours Risk (=Prob.
That Actual PH Will Exeed XAxis Value)
Systems Engineering Person Hours Risk
100%
90%
y = -1E-15x3 + 3E-10x2 - 3E-05x + 1.1888
80%
70%
2
R = 0.9635
60%
50%
40%
30%
20%
10%
0%
0
20000
40000
60000
80000
100000
120000
140000
Person Hours
Use this Graph to help understand labor (and cost) exposure for your choice
of labor (say to be bid in a proposal)
Example: The risk of exceeding 6000 labor hours is slightly less than 20%.
Note: You might choose to present only the smooth curve fit rather than the discrete plot.
9
Summary COSYSMOR Person Hours/ Person Months
and Schedule Risk/Confidence Statistics
Item
Effort
Person Hours
Minimum =
5679
Risk=
99.37%
Confidence=
0.63%
Most Likely=
28361
Risk=
37.50%
Confidence=
62.50%
Maximum =
132952
Risk=
0.00%
Confidence=
100.00%
20% Risk/ 80%
Confidence=
Ideal
Person Months*
Schedule **
37.4
8.3
186.6
14.0
874.7
23.4
42606
280.3
16.1
33538
220.6
14.8
50% Risk/ 50%
Confidence=
28361
186.6
14.0
95% Risk/5%
Confidence =
8024
52.8
9.3
5% Risk/95%
Confidence =
104246
685.8
21.6
30% Risk/ 70%
Confidence=
152
* Person Hours Per Person Month
This table summarizes the data from the effort risk and the “ideal” schedule risk
estimates.
10
Practical Software and
Systems Measurement
A foundation for objective project management
PSM
2007: The Breakout
Year for COSYSMO
Wed-Thurs (7/25-26)
John Rieff/John Gaffney/Garry Roedler
PSM Users Group Conference
23-27 July 2007
Golden, Colorado
11
Next Steps/Action Items from COSYSMO
Workshop at PSM User Group Conference
• Agreed to Expand COSYSMO to cover Risk and
Reuse  i.e., add to COSYSMO baseline
• Peer Review the User’s Guide by mid-Sept
• Harmonize “COCOMO Suite” of estimation
models/tools for Systems Engineering/System of
Systems/Software Cost Estimation
• Clarify system hierarchies as they relate to size driver
counts, levels of recursion and unit effort
• Verify that any proposed changes do not invalidate
early calibration efforts
12
For Further Information, Contact:
•
John Gaffney
j.gaffney@lmco.com
•
301-721-5710
Garry Roedler
garry.j.roedler@lmco.com 610-354-3625
856-792-9406
13
Backup
14
Concerning Risk Distributions
•
•
•
•
COSYSMOR provides “risk” and “confidence” distributions for the labor and
schedule or project duration estimates, based three-point values for each of
its parameters that the user enters.
Risk=Prob[actual value >target value]; the complementary cumulative
distribution function (CCDF).
Confidence=100%-Risk%=Prob[actual≥ target value]; the cumulative
distribution function (CDF) of the cost.
Note: these definitions apply to quantities for which “better” is smaller, e.g.,
effort/cost and project duration. They are reversed for cases in which
“better” is larger, such as Mean-Time-Between Failure.
The COSYSMOR risk assessment capability is implemented using threepoint approximations; they are non-parametric, meaning that they are not
derived as approximations to any particular distribution such as a Gamma
or a Weibull.
– This in contrast to the use of Monte Carlo methods, in which a particular
distribution is used and then a large number of instances are generated from it.
•
COSYSMOR does not generate such a large number of instances.
– Rather, it generates an approximation to the distribution from the 3 point
approximations to each variable. For example, if there are 4 (mutually
independent) variables, the approximation has 81 values (=3x3x3x3).
15
Probability Approximation Used In
COSYSMOR
•
COSYSMOR implements the approximation developed by Keefer and
Bodily, the “extended Pearson-Tukey” method.
– They evaluated 22 approximations, and found this one to be the best in
terms of their abilities to estimate the means and variances of various
distributions.
•
This method approximates a continuous distribution by a discrete one:
Fractile
Probability Assigned
0.05
0.185
0.50
0.630
0.95
0.185
16