Proxy Estimation Costing
for Systems (PECS)
October 2012
Reggie Cole
Lockheed Martin Senior Fellow
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
Discussion Topics
 Why Do We Need Yet Another Cost Model?
– The gap in early-stage system cost modeling
 Systems Engineering Effort as a Proxy Estimator for System Cost
– And the role of COSYSMO is arriving at this proxy estimate
2
 Proxy Estimation Costing for Systems (PECS)
– Derivation of the PECS Model
– The PECS modeling approach
 Case Study for Affordability Analysis Using the PECS Model
– The real power of the PECS model
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
Cost Modeling Needs Change Over Time in Terms of
Speed and Accuracy – So Does Solution Information
Detailed
Solution
Description
We Have a Good Selection
of Tools for Late-Stage
Cost Modeling
High-Level
Solution
Description
Cost Estimate
± 5%
Cost Estimate
± 10%
3
Increasingly
Refined
Information
About the
Solution
High-Level
Solution
Assumptions
Problem-Space
Description
Cost Estimate
± 20%
Increasingly
Refined Cost
Estimate
Cost Estimate
± 25%
We Have Gaps in
Early-Stage Cost
Modeling
Increasing Effort and Cost-Modeling Expertise
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
Systems Engineering Effort as a Proxy Measure of
Overall System Size and Complexity
 Proxy Measures
– Proxy measures are used when you cannot directly measure what you
want to measure – and when an indirect measure provides sufficient insight
– Proxy measures are often used in clinical studies since direct measurement
is often infeasible or can even alter the outcome
– It is not always possible to directly measure what you want to measure – or
directly estimate what you want to estimate
 System Engineering Effort is a Proxy Measure for System Cost
– There is strong evidence for the link between systems engineering effort
and program cost – dating back to a NASA study in the 1980s
– The optimal relationship between systems engineering effort and overall
program cost is 10% - 15%
– Industry has long used a parametric relationship between software cost
and systems engineering cost for software-intensive systems
– Systems engineering effort can be an effective proxy measure for overall
system cost
H. Dickinson, S. Hrisos, M. Eccles, J. Francis, M. Johnston, Statistical Considerations in a Systematic Review of Proxy Measures of Clinical
Behaviour, Implementation Science, 2010
E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
4
COSYSMO 2.0 Model Parameters Provide a Rich
Assessment of System Size, Complexity and Reuse
Size Drivers
Number of System Requirements
Number of Major System Interfaces
Initial Estimate of
System Size
Reuse Factors
Managed Elements
Number of Critical Algorithms
Adopted Elements
Number of Operational Scenarios
Deleted Elements
Modified Elements
Cost Drivers
New Elements
Requirements Understanding
Architecture Understanding
Scaled Estimate
of System Size
Level of Service Requirements
5
Migration Complexity
Technology Risk
Level of Documentation Required
Diversity of Installed Platforms
Consolidated Cost
Driver Factor
Level of Design Recursion
Stakeholder Team Cohesion
Personnel / Team Capability
Personnel Experience / Continuity
Process Capability
Multisite Coordination
Level of Tool Support
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
Estimate of Systems Engineering
Effort…Also a Biased Proxy
Estimator for System
Scope…And System Cost
An Approach for De-Biasing the Proxy Estimator –
Relationship Between SE Effort and Total Effort
Total Program Overrun
32 NASA Programs
200
Definition $
Definition Percent = ---------------------------------Target + Definition$
Program Overrun
180
160
NASA data supports a 10%-15% optimal
allocation of systems engineering effort
as a portion of overall program effort
Actual + Definition$
Program Overrun = ---------------------------------Target + Definition$
140
120
100
80
60
40
R2 = 0.5206
20
0
0
5
10
15
20
Definition Percent of Total Estimate
6
3.0
Actual/Planned Cost
2.6
INCOSE study on the value of systems
engineering also supports a 10%-15%
optimal allocation of systems engineering
as a portion of overall program effort
2.2
1.8
1.4
1.0
0%
4%
8%
12%
16%
20%
24%
0.6
SE Effort = SE Quality * SE Cost/Actual Cost
28%
W. Gruhl, Lessons Learned, Cost/Schedule Assessment Guide,”
Internal Presentation, NASA Comptroller’s Office, 1992
E. Honour, “Understanding the Value of Systems Engineering,”
INCOSE, 2004
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
The PECS Cost Function
CostSystem  EffortSE
FCal RateLabor
 CostMaterials  CostTravel
FConv
Where:
FCal : COSYSMO Calibration Factor(Deterministic)
FConv : Factorfor ConvertingSE Effort toT otalP rogramEffort(Stochastic)
EffortSE : SE EffortComputedUsing COSYSMO (Stochastic)
RateLabor : Labor Rate (Stochastic)
CostMaterials : MaterialCosts (Stochastic)
CostTravel : T ravelCosts (Stochastic)
7
Variable
Type
Description
COSYSMO Calibration Factor
Deterministic Scalar Value
Organization-specific calibration factor
Effort Conversion Factor
Triangular Distributed Random Variable
Three-point estimate of factor to convert SE effort to total
program effort (nominally 0.08, 0.12 and 0.16)
SE Effort
Triangular Distributed Random Variable
Three-point estimate for SE effort, generated using COSYSMO
Labor Rate
Triangular Distributed Random Variable
Three-point estimate for composite labor rate
Material Costs
Triangular Distributed Random Variable
Three-point estimate for material costs
Travel Costs
Triangular Distributed Random Variable
Three-point estimate for travel costs
This Model is Well Positioned for Monte Carlo Analysis
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
The PECS Model – Putting It All Together
Estimator Bias Function is Based on the
Well-Established Relationship Between
SE Effort and Overall Program Effort
Proxy Estimation Costing for Systems (PECS)
SE Effort is an estimator
for total system cost…but
it is a biased estimator
 Customer Requirements
 System Interfaces
 Major Algorithms
 Operational Scenarios
 Complexity Drivers (Problem/Solution)
 Requirements Understanding
 Architecture Understanding
 Level of Service Requirements
 Migration Complexity
 Technology Risk
 Documentation Needs
 Installations/Platform Diversity
 Levels of Recursion in the Design
Stakeholder Team Cohesion
 Personnel/Team Capability
 Personnel Experience/Continuity
 Process Capability
 Multisite Coordination
 Tool Support
2.6
Actual/Planned Cost
Size Drivers (Problem Space)
3.0
2.2
1.8
1.4
1.0
0%
Estimator
De-Biasing
4%
8%
16%
20%
24%
28%
8
0.6
SE Effort = SE Quality * SE Cost/Actual Cost
CostSystem  EffortSE
 Reuse Factors (Solution Space)
 New
 Modified
 Deleted
 Adopted
 Managed
12%
FCal RateLabor
 CostMaterials  CostTravel
FConv
40.00
Monte Carlo Analysis of System Cost
35.00
Cost Estiate ($M)
30.00
25.00
20.00
15.00
10.00
5.00
0.00
1
4
7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100
Confidence
Three different COSYMO scenarios – optimistic,
expected & pessimistic – provide the basis for the
Monte Carlo analysis of system cost
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
Case Study – The COSYSMO Scenarios
The case study is based on a large C2 system. Initially developed 20 years ago, the system was
unprecedented. Twenty years later, a replacement system is needed. While the initial development
was unprecedented, the replacement system is not, which drives down the size drivers (through
reuse) and cost drivers.
The case study looks at three cost scenarios:
 Case 1 – The original unprecedented system (for calibration purposes)
 Case 2 – Replacement system (as a new development)
 Case 3 – Replacement system (as a largely COTS/GOTS approach)
9
COSYSMO Scenarios for PECS – Three Scenarios for Each Case
Case 2 - Replacement System (Developed)
1000
4.00
500
2.00
0
0.00
Pessimistic
Expected
Optimistic
0.60
800
600
0.40
400
0.20
200
0
0.00
Pessimistic
Expected
Optimistic
600
0.6
500
0.5
400
0.4
300
0.3
200
0.2
100
0.1
0
0
Pessimistic
Expected
Optimistic
Requirements
System I/F
Requirements
System I/F
Requirements
System I/F
Algorithms
Scenarios
Algorithms
Scenarios
Algorithms
Scenarios
Cost Driver Factor
Cost Driver Factor
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
Cost Driver Factor
Cost Driver Factor
6.00
0.80
1000
Size (Effective Requirements)
8.00
1500
1200
Case 3 - Replacement System (COTS/GOTS)
Cost Driver Factor
10.00
Size (Effective Requirements)
2000
Cost Driver Factor
Size (Effective Requirements)
Case 1 - Large Unprecedented System
Case Study – The Monte Carlo Analysis
Case 1
Average 80/20 Cost = $1.9B
Used as a calibration point
for the model
Case 1 - Large Unprecedented System
3000.00
Cost Estiate ($M)
2500.00
2000.00
1500.00
Case 2
Average 80/20 Cost = $77M
Initial Solution for
Replacement System
1000.00
Case 2 - Replacement System (Development)
500.00
140.00
0.00
1
4
120.00
7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100
Cost Estiate ($M)
Confidence
100.00
80.00
60.00
10
40.00
Case 3 - Repacement System (COTS/GOTS)
20.00
50.00
0.00
1
4
7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 45.00
61 64 67 70 73 76 79 82 85 88 91 94 97 100
Confidence
40.00
Case 3
Average 80/20 Cost = $30M
More Affordable Solution,
Based on COTS/GOTS Solution
Cost Estiate ($M)
35.00
30.00
25.00
20.00
15.00
10.00
5.00
0.00
1
4
7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100
Confidence
The PECS Model Enables Rapid Turn-Around Analysis of
Alternatives and “Should Cost” Analysis
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
Conclusion
 The PECS Model is Based on Well-Established Approaches
– COSYSMO provides the basis for estimation of systems engineering effort
– and a biased proxy estimator for overall system cost
– There is a well-established relationship between systems engineering effort
and overall effort used to de-bias the COSYSMO-modeled effort
– Monte Carlo analysis is a well-established technique for cost modeling
 The PECS Model Can Improve System Cost Modeling
– The PECS Model occupies an important niche – fully parametric system
cost modeling in the early stages of system definition
– The PECS Model can serve as a powerful affordability analysis tool –
supporting rapid-turnaround analysis of alternatives
– But…the PECS Model is not a replacement for existing models
 Next Steps
– Broader validation of the model
– Cross-industry review of the model
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.
11
12
Example © 2012 Lockheed Martin Corporation. All Rights Reserved.