advertisement

Proxy Estimation Costing for Systems (PECS) October 2012 Reggie Cole Lockheed Martin Senior Fellow Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Discussion Topics Why Do We Need Yet Another Cost Model? – The gap in early-stage system cost modeling Systems Engineering Effort as a Proxy Estimator for System Cost – And the role of COSYSMO is arriving at this proxy estimate 2 Proxy Estimation Costing for Systems (PECS) – Derivation of the PECS Model – The PECS modeling approach Case Study for Affordability Analysis Using the PECS Model – The real power of the PECS model Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Cost Modeling Needs Change Over Time in Terms of Speed and Accuracy – So Does Solution Information Detailed Solution Description We Have a Good Selection of Tools for Late-Stage Cost Modeling High-Level Solution Description Cost Estimate ± 5% Cost Estimate ± 10% 3 Increasingly Refined Information About the Solution High-Level Solution Assumptions Problem-Space Description Cost Estimate ± 20% Increasingly Refined Cost Estimate Cost Estimate ± 25% We Have Gaps in Early-Stage Cost Modeling Increasing Effort and Cost-Modeling Expertise Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Systems Engineering Effort as a Proxy Measure of Overall System Size and Complexity Proxy Measures – Proxy measures are used when you cannot directly measure what you want to measure – and when an indirect measure provides sufficient insight – Proxy measures are often used in clinical studies since direct measurement is often infeasible or can even alter the outcome – It is not always possible to directly measure what you want to measure – or directly estimate what you want to estimate System Engineering Effort is a Proxy Measure for System Cost – There is strong evidence for the link between systems engineering effort and program cost – dating back to a NASA study in the 1980s – The optimal relationship between systems engineering effort and overall program cost is 10% - 15% – Industry has long used a parametric relationship between software cost and systems engineering cost for software-intensive systems – Systems engineering effort can be an effective proxy measure for overall system cost H. Dickinson, S. Hrisos, M. Eccles, J. Francis, M. Johnston, Statistical Considerations in a Systematic Review of Proxy Measures of Clinical Behaviour, Implementation Science, 2010 E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. 4 COSYSMO 2.0 Model Parameters Provide a Rich Assessment of System Size, Complexity and Reuse Size Drivers Number of System Requirements Number of Major System Interfaces Initial Estimate of System Size Reuse Factors Managed Elements Number of Critical Algorithms Adopted Elements Number of Operational Scenarios Deleted Elements Modified Elements Cost Drivers New Elements Requirements Understanding Architecture Understanding Scaled Estimate of System Size Level of Service Requirements 5 Migration Complexity Technology Risk Level of Documentation Required Diversity of Installed Platforms Consolidated Cost Driver Factor Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Multisite Coordination Level of Tool Support Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Estimate of Systems Engineering Effort…Also a Biased Proxy Estimator for System Scope…And System Cost An Approach for De-Biasing the Proxy Estimator – Relationship Between SE Effort and Total Effort Total Program Overrun 32 NASA Programs 200 Definition $ Definition Percent = ---------------------------------Target + Definition$ Program Overrun 180 160 NASA data supports a 10%-15% optimal allocation of systems engineering effort as a portion of overall program effort Actual + Definition$ Program Overrun = ---------------------------------Target + Definition$ 140 120 100 80 60 40 R2 = 0.5206 20 0 0 5 10 15 20 Definition Percent of Total Estimate 6 3.0 Actual/Planned Cost 2.6 INCOSE study on the value of systems engineering also supports a 10%-15% optimal allocation of systems engineering as a portion of overall program effort 2.2 1.8 1.4 1.0 0% 4% 8% 12% 16% 20% 24% 0.6 SE Effort = SE Quality * SE Cost/Actual Cost 28% W. Gruhl, Lessons Learned, Cost/Schedule Assessment Guide,” Internal Presentation, NASA Comptroller’s Office, 1992 E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004 Example © 2012 Lockheed Martin Corporation. All Rights Reserved. The PECS Cost Function CostSystem EffortSE FCal RateLabor CostMaterials CostTravel FConv Where: FCal : COSYSMO Calibration Factor(Deterministic) FConv : Factorfor ConvertingSE Effort toT otalP rogramEffort(Stochastic) EffortSE : SE EffortComputedUsing COSYSMO (Stochastic) RateLabor : Labor Rate (Stochastic) CostMaterials : MaterialCosts (Stochastic) CostTravel : T ravelCosts (Stochastic) 7 Variable Type Description COSYSMO Calibration Factor Deterministic Scalar Value Organization-specific calibration factor Effort Conversion Factor Triangular Distributed Random Variable Three-point estimate of factor to convert SE effort to total program effort (nominally 0.08, 0.12 and 0.16) SE Effort Triangular Distributed Random Variable Three-point estimate for SE effort, generated using COSYSMO Labor Rate Triangular Distributed Random Variable Three-point estimate for composite labor rate Material Costs Triangular Distributed Random Variable Three-point estimate for material costs Travel Costs Triangular Distributed Random Variable Three-point estimate for travel costs This Model is Well Positioned for Monte Carlo Analysis Example © 2012 Lockheed Martin Corporation. All Rights Reserved. The PECS Model – Putting It All Together Estimator Bias Function is Based on the Well-Established Relationship Between SE Effort and Overall Program Effort Proxy Estimation Costing for Systems (PECS) SE Effort is an estimator for total system cost…but it is a biased estimator Customer Requirements System Interfaces Major Algorithms Operational Scenarios Complexity Drivers (Problem/Solution) Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Documentation Needs Installations/Platform Diversity Levels of Recursion in the Design Stakeholder Team Cohesion Personnel/Team Capability Personnel Experience/Continuity Process Capability Multisite Coordination Tool Support 2.6 Actual/Planned Cost Size Drivers (Problem Space) 3.0 2.2 1.8 1.4 1.0 0% Estimator De-Biasing 4% 8% 16% 20% 24% 28% 8 0.6 SE Effort = SE Quality * SE Cost/Actual Cost CostSystem EffortSE Reuse Factors (Solution Space) New Modified Deleted Adopted Managed 12% FCal RateLabor CostMaterials CostTravel FConv 40.00 Monte Carlo Analysis of System Cost 35.00 Cost Estiate ($M) 30.00 25.00 20.00 15.00 10.00 5.00 0.00 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 Confidence Three different COSYMO scenarios – optimistic, expected & pessimistic – provide the basis for the Monte Carlo analysis of system cost Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Case Study – The COSYSMO Scenarios The case study is based on a large C2 system. Initially developed 20 years ago, the system was unprecedented. Twenty years later, a replacement system is needed. While the initial development was unprecedented, the replacement system is not, which drives down the size drivers (through reuse) and cost drivers. The case study looks at three cost scenarios: Case 1 – The original unprecedented system (for calibration purposes) Case 2 – Replacement system (as a new development) Case 3 – Replacement system (as a largely COTS/GOTS approach) 9 COSYSMO Scenarios for PECS – Three Scenarios for Each Case Case 2 - Replacement System (Developed) 1000 4.00 500 2.00 0 0.00 Pessimistic Expected Optimistic 0.60 800 600 0.40 400 0.20 200 0 0.00 Pessimistic Expected Optimistic 600 0.6 500 0.5 400 0.4 300 0.3 200 0.2 100 0.1 0 0 Pessimistic Expected Optimistic Requirements System I/F Requirements System I/F Requirements System I/F Algorithms Scenarios Algorithms Scenarios Algorithms Scenarios Cost Driver Factor Cost Driver Factor Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Cost Driver Factor Cost Driver Factor 6.00 0.80 1000 Size (Effective Requirements) 8.00 1500 1200 Case 3 - Replacement System (COTS/GOTS) Cost Driver Factor 10.00 Size (Effective Requirements) 2000 Cost Driver Factor Size (Effective Requirements) Case 1 - Large Unprecedented System Case Study – The Monte Carlo Analysis Case 1 Average 80/20 Cost = $1.9B Used as a calibration point for the model Case 1 - Large Unprecedented System 3000.00 Cost Estiate ($M) 2500.00 2000.00 1500.00 Case 2 Average 80/20 Cost = $77M Initial Solution for Replacement System 1000.00 Case 2 - Replacement System (Development) 500.00 140.00 0.00 1 4 120.00 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 Cost Estiate ($M) Confidence 100.00 80.00 60.00 10 40.00 Case 3 - Repacement System (COTS/GOTS) 20.00 50.00 0.00 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 45.00 61 64 67 70 73 76 79 82 85 88 91 94 97 100 Confidence 40.00 Case 3 Average 80/20 Cost = $30M More Affordable Solution, Based on COTS/GOTS Solution Cost Estiate ($M) 35.00 30.00 25.00 20.00 15.00 10.00 5.00 0.00 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97 100 Confidence The PECS Model Enables Rapid Turn-Around Analysis of Alternatives and “Should Cost” Analysis Example © 2012 Lockheed Martin Corporation. All Rights Reserved. Conclusion The PECS Model is Based on Well-Established Approaches – COSYSMO provides the basis for estimation of systems engineering effort – and a biased proxy estimator for overall system cost – There is a well-established relationship between systems engineering effort and overall effort used to de-bias the COSYSMO-modeled effort – Monte Carlo analysis is a well-established technique for cost modeling The PECS Model Can Improve System Cost Modeling – The PECS Model occupies an important niche – fully parametric system cost modeling in the early stages of system definition – The PECS Model can serve as a powerful affordability analysis tool – supporting rapid-turnaround analysis of alternatives – But…the PECS Model is not a replacement for existing models Next Steps – Broader validation of the model – Cross-industry review of the model Example © 2012 Lockheed Martin Corporation. All Rights Reserved. 11 12 Example © 2012 Lockheed Martin Corporation. All Rights Reserved.