Some Experience With
COSYSMOR At Lockheed Martin
John Gaffney
301-721-5710
j.gaffney@lmco.com
Lockheed Martin Systems & Software Resource Center (SSRC)
Center for Process Improvement Excellence (CPIE)
October 29, 2008
(c) Copyright, Lockheed Martin
Corporation, 2008
1
Agenda
• What Is COSYSMOR ?
• Some Experience With COSYSMOR
(c) Copyright, Lockheed Martin
Corporation, 2008
2
COSYSMOR or COSYSMO Risk and Reuse
• COSYSMOR evolved from the USC Academic
COSYSMO systems engineering estimation model/tool
(Dr. Ricardo Valerdi) and LMCO implemented these
enhancements in a tool, COSYSMOR, or COSYSMO
Risk
• COSYSMOR is available on request
– It has been distributed to some organizations outside of
Lockheed Martin already
• A major driver for the development of COSYSMOR was
to get away from “single point” cost estimates in order to
better recognize the uncertainty associated with effort,
schedule, and cost estimates
(c) Copyright, Lockheed Martin
Corporation, 2008
3
Additional Functions Provided By COSYSMOR
COSYSMOR provides four major additional functions
beyond those provided by Academic COSYSMO:
1.
Estimation of Cost/Effort and Schedule Uncertainties/Risk and
Confidence: Provides quantification of the impacts of
uncertainties in the values of key model parameter values.
Provides multiple cost and schedule values with associated
probabilities.
Risk=Prob [Actual Effort Will Be >Estimated Effort]
Confidence=100%-Risk
2.
3.
4.
Representation of Multiple Types of Size Drivers: Provides for
entering counts of: new, modified, reused, and deleted types for
each of the four size driver categories.
Labor Scheduling: Provides the spread of systems engineering
labor for the five systems engineering activities and across four
the development phases (time).
Labor Allocation: Provides for the user to select the percentage
allocations of the twenty activity/phase pairs or effort elements.
(c) Copyright, Lockheed Martin
Corporation, 2008
4
What Lockheed Martin Has Been Doing
• Been calibrating COSYSMOR in various organizations
and applying it in several projects
• Have also developed an analogous model/tool for
software resource estimation, COCOMOR
• We are quite interested in harmonizing the estimation of
systems engineering effort and software engineering
effort
• We have also developed a tailored version of
COSYMOR, COSYSMOR-A for avionics
– This illustrates the general utility of this model/tool form
(c) Copyright, Lockheed Martin
Corporation, 2008
5
COSYSMOR-A
• COSYSMOR-A stands for COSYSMOR for Avionics
• COSYSMOR-A is a tailored version of COSYSMOR, to
produce resource (effort) estimates for new avionics
systems and upgrades produced by Lockheed Martin
Aeronautics Company Mission Systems and Avionics
• The tailoring included changes in two of the size drivers
and two of the cost drivers (of COSYSMO/COSYSMOR):
– Size Drivers: # of Hardware Components and # of Software
Functions replace # of Algorithms and # of Operational
Scenarios in COSYSMO/R, respectively
– Cost Drivers: Lab System Integration and Test Complexity and
Flight Test Complexity replace Migration Complexity and
Multisite Coordination in COSYSMO/R, respectively
(c) Copyright, Lockheed Martin
Corporation, 2008
6
COSYSMOR Data Entry-1
ENTER SIZE PARAMETERS FOR SYSTEM OF INTEREST
Low
Likely*
Easy
# of System Requirements
# of System Interfaces
# of Algorithms
# of Operational Scenarios
Equivalent Total Requirements Size
Equivalent New Requirements Size
Requirements Understanding
Architecture Understanding
Level of Service Requirements
Migration Complexity
Technology Risk
Documentation
# and diversity of installations/platforms
# of recursive levels in the design
Stakeholder team cohesion
Personnel/team capability
Personnel experience/continuity
Process capability
Multisite coordination
Tool support
Composite effort multipliers
Nominal
Difficult
Easy
Nominal
Difficult
10
1
10
11
2
10
3
2
11
4
9
2
4
10
3
5
4
2
5
5
351.90
402.90
299.58
341.39
SELECT COST PARAMETERS FOR SYSTEM OF INTEREST
9
2
3
4
VH
H
L
N
L
L
N
VL
H
H
VH
VH
VH
H
Low Value
0.60
0.81
0.79
1.00
0.84
0.91
1.00
0.80
0.81
0.81
0.67
0.77
0.80
0.85
0.05
N
N
N
N
N
N
N
N
N
N
N
N
N
N
Likely Value*
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
L
L
N
EH
H
VH
EH
H
VL
VL
L
L
L
L
High Value
1.36
1.27
1.00
1.92
1.32
1.28
1.86
1.21
1.50
1.48
1.21
1.21
1.15
1.16
54.58
High
Easy
Nominal
11
2
5
4
Difficult
12
13
11
6
1
5
7
6
520.40
437.77
Size Parameter Range Values *
* Note: If you do not wish to use the range or uncertainty
for the size and cost parameters, but wish to just enter a
then enter such values in the "Likely" size parameter an
simply ignore the range estimates produced or make the
equal to the Likely value.
** You can specify the proportions of each of the four re
their cost per requirement relative to that for a new requ
you can enter your estimate in the "Requirements Types
category, e.g., "Systems Requirements", that is "Modifi
these three data entries automatically. Also, enter the "R
of each requirements category. The count of "Deleted" i
the course of execution of a project might well have som
EAC (Estimate-At-Completion).The "Relative Cost" is the
figure were 10%, that would mean that that type, e.g., "R
Three-point data entry for size and cost drivers.
(c) Copyright, Lockheed Martin
Corporation, 2008
7
COSYSMOR Data Entry-2
• You indicate the uncertainty in size driver and cost driver
values by entering Low, Likely and High values for each
to indicate your assessment of the distribution for the
easy, nominal and difficult size driver counts
– If you don’t believe that there is a range, but rather, there is the
value, then you enter the same values for Low, Likely and High
(c) Copyright, Lockheed Martin
Corporation, 2008
8
COSYSMOR Data Entry-3
Requirements Types **
New
Proportion, pN
Calculated
62.00%
71.00%
66.00%
71.00%
Estimated Selected
75.00%
75.00%
75.00%
75.00%
Modified
Proportion, pM
75.00%
75.00%
75.00%
75.00%
Reused
Proportion,pR
Deleted
Proportion,pD
Rel. Cost*** Calculated Estimated Selected Rel Cost*** Calculated Estimated Selected
Rel Cost ***
Calculated Estimated Selected
Rel Cost ***
100.00%
5.00%
0.00%
0.00%
78.16%
30.00%
25.00%
25.00%
54.43%
3.00%
0.00%
0.00%
10.72%
100.00%
2.00%
0.00%
0.00%
78.16%
25.00%
25.00%
25.00%
54.43%
2.00%
0.00%
0.00%
10.72%
100.00%
4.00%
0.00%
0.00%
78.16%
27.00%
25.00%
25.00%
54.43%
3.00%
0.00%
0.00%
10.72%
100.00%
3.00%
0.00%
0.00%
78.16%
24.00%
25.00%
25.00%
54.43%
2.00%
0.00%
0.00%
10.72%
•A Program User enters the percents of Modified, Reused and Deleted for
each of the four size drivers; the tool computes the percent of New
•These percents are taken to apply to the Easy, Nominal and Difficult
values for each of the four types of size drivers
•A Program User also enters the Relative Cost (relative to New) for the
Modified, Reused and Deleted counts
Note: If the Program User wants to designate different sets of percents and relative costs for Easy,
Nominal and Difficult, then he will have to go the Reuse page of COSYSMOR to enter these values.
(c) Copyright, Lockheed Martin
Corporation, 2008
9
COSYSMOR Data Entry-4
COSYSMORA MODEL PARAMETERS
Equivalent Size, S (=Equivalent New)
Unit Effort Constant, A:Baseline
Unit Effort Constant, A:User**
Unit Effort Constant, A Selected
Size Exponent, E
Cost Parameter Product,D
SYSTEMS ENGINEERING PERSON MONTHS
SYSTEMS ENGINEERING PERSON HOURS
Low
1979
39.100
Likely
1979
39.800
49.837
39.100
1.000
0.364
182.7
28134
50.729
39.800
1.000
1.202
614.8
94684
High
Nominal *
1979
1979
40.080
39.48
51.086
40.080
1.000
3.752
1932.1
297540
39.48
39.48
1.000
1.00
507.2
78115
An Administrative User enters the Low, Likely and High values for A, the Unit
Effort Constant, and E, the Exponent, in the COSYSMOR model:
K=A*SE* Di
(c) Copyright, Lockheed Martin
Corporation, 2008
10
Some COSYSMOR Outputs
Principal outputs of the COSYSMOR tool are:
1.
2.
3.
4.
5.
Equivalent Size Risk
Cost Driver Product Risk
Systems Engineering Person Hours Risk
Systems Engineering Person Hours Overrun Risk
Percent of Total Systems Engineering Effort By
Process Activity and By Phase
6. Systems Engineering Person Hours Vs. Month
7. Systems Engineering Schedule Risk
(c) Copyright, Lockheed Martin
Corporation, 2008
11
Systems Engineering Person Hours Risk
Person Hours Risk (=Prob. That
Actual PH Will Exeed X-Axis
Value)
Systems Engineering Person Hours Risk
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
0
20000
40000
60000
80000
100000
120000
140000
Person Hours
Use this Graph to help understand labor (and cost) exposure for your choice
of labor (say to be bid in a proposal). Example: The risk of exceeding 60000 labor
hours is slightly less than 20%. Note: You might choose to present only the
smooth curve fit rather than the discrete plot.
(c) Copyright, Lockheed Martin
Corporation, 2008
12
Systems Engineering Person Hours Overrun Risk
Person Hours Overrun Risk For Target Person Hours= 250000
Person Hours Overrun Risk (=Prob That
Target PH Value Will Be Exceeded)
50%
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
Person Hours Overrun
This plot shows the overrun risk % for a target of 250000 of Systems Engineering Person Hours
values. This plot is the tail of the Systems Engineering Person Hours Risk plot. Some persons
might prefer to see this plot in which the “zero” value on the horizontal axis corresponds to the
target value and the horizontal axis values are the (potential) overrun values relative to the target.
(c) Copyright, Lockheed Martin
Corporation, 2008
13
(c) Copyright, Lockheed Martin
Corporation, 2008
14
Observations
• Better management of uncertainty starts with recognizing
its existence and in estimating cost/schedule risk
• Use the risk information to make better management
decisions during both business acquisition and program
execution
– Serve as a basis for discussion of alternatives within the project
as well as with the customer
– A key aspect of affordability
• Relate cost/schedule risk to risk register as part of
normal risk management process
– Relate to potential program failure modes
• Need to harmonize SE and SW estimation
(c) Copyright, Lockheed Martin
Corporation, 2008
15