University of Southern California Center for Systems and Software Engineering USC CSSE 2012 Annual Research Review Mathematical Formulation and Validation of the Impact of Requirements Volatility on Systems Engineering Effort March 6, 2012 Mauricio E. Peña 1 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Outline • Motivation and introduction • Research methods • Observations from the research • Model Development • Results – Model Calibration – Evaluation of Model Performance – Sensitivity Analysis – Cross-Validation • Conclusion 2 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Importance of Understanding Requirements Volatility • Requirements volatility has been identified by numerous research studies as a risk factor and cost-driver of systems engineering projects1 • Requirements changes are costly, particularly in the later stages of the lifecycle process because the change may require rework of the design, verification and deployment plans2 • The Government Accountability Office (GAO) concluded in a 2004 report on the DoD’s acquisition of software-intensive weapons systems that missing, vague, or changing requirements are a major cause of project failure3 System developers often lack effective methods and tools to account for and manage requirements volatility Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393 3 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Requirements Volatility is Expected • Changes to requirements are a part of our increasingly complex systems & dynamic business environment – Stakeholders needs evolve rapidly – The customer may not be able to fully specify the system requirements up front – New requirements may emerge as knowledge of the system evolves – Requirements often change during the early phases of the project as a result of trades and negotiations Requirements volatility must be anticipated and managed Sources: Kotonya and Sommerville (1995); Reifer (2000) 4 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review CSSE Parametric Cost Models • The Constructive Systems Engineering Cost Model (COSYSMO) was developed by the USC Center for Software and Systems Engineering (CSSE) in collaboration with INCOSE and Industry affiliates • COSYSMO is the first generally-available parametric cost model designed to estimate Systems Engineering effort • Built on experience from COCOMO 1981, COCOMO II • During the development of COSYSMO, volatility was identified as a relevant adjustment factor to the model’s size drivers Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm 5 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review COSYSMO Operational Concept Volatility Factor # Requirements # Interfaces # Scenarios # Algorithms Size Drivers Effort Multipliers - Application factors -8 factors - Team factors -6 factors Source: Valerdi (2005) COSYSMO Effort Calibration 6 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review COSYSMO Cost Estimating Relationships (CERs) n SE _ Hrs A SIZE ( EM i ) E i 1 SE_Hrs = Systems Engineering effort (hours) A = calibration constant derived from historical project data SIZE = measure of functional size of the system (number of requirements, interfaces, algorithms, operational scenarios) n = number of cost drivers (14) EM = effort multiplier for the ith cost driver The exponent (E) accounts for diseconomies of scale Source: Valerdi (2005) 7 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Research Methods USC University of Southern California C S E Center for Software Engineering Analyze Existing literature 1 Literature Review & 6 Workshops completed Data gathered from 25 industry projects Perform Behavioral Analysis 2 Identify Relative Significance 3 A-PRIORI MODEL + SAMPLING DATA = A-POSTERIORI MODEL Perform ExpertJudgement, Delphi Assessment 4 Gather Project Data We are here 5 Determine Bayesian A-Posteriori Update 6 Gather more data; refine model 7 USC-CSE Annual Research Review – 3/17/03 Source: Boehm et al (2000) 8 8 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Requirements Volatility Observations 1. Requirements volatility is caused by an identifiable set of project and organizational factors 2. The level of requirements volatility is a function of the system life cycle phase 3. Requirements volatility leads to an increase in project size and cost 4. The cost and effort impact of a requirements change increases the later the change occurs in the system life cycle 5. The impact of requirements volatility may vary depending on the type of change: added, deleted, or modified 9 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Model Development ( 1 of 3) • Incorporated volatility effects through a scale factor (SF) added to the diseconomies of scale exponent (E) n PM E SF A SIZE ( EM i ) i 1 • Similar approach used to model volatility effects in Ada COCOMO1 • Prior research points to the compounding or exponential effect of project factors with variable life cycle impact2 Source: 1: Boehm, B. and Royce, W. (1989); 2: Wang, G. et al., (2008) 10 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Model Development (2 of 3) • In Ada COCOMO, the requirements volatility scale factor SF1 was rated using range of 0.00 to 0.05, with higher values indicating a greater volatility • Utilizing a similar methodology, SF is calculated using a constant value of 0.05 that is scaled based on REVL, the % of the baseline requirements that is expected to change over the system lifecycle REVL SF 0.05 100 Expected REVL is rated as Very Low, Low, Moderate, High and Very High 11 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Model Development (3 of 3) • Observations #2 and #4 indicate that both the level and impact of requirements volatility change as a function of time • A weighting factor (wv) is added to account for these additional effects REVL SFw 0.05 wvl 100 • Where wv1 wl l l – wvl = aggregate lifecycle phase volatility weighting factor – wl = weighting factor for each life cycle phase – Θl = % of total requirements changes per life cycle phase 12 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Model Calibration A Priori Model Initial parameter mean and variance Historical data Data-determined Model Regression Analysis; OLS Mean and Variance Expert-judgment estimates A Posteriori Model Bayesian Analysis Formally combines expert judgment With sample data Source: Boehm et al. (2000); Nguyen (2010) Updated parameter mean and variance Optimized combination of data sources to increase precision 13 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Data Collection • • Collected expert judgment on REVL and weighting factors through surveys and workshops – Software and Systems Engineers with 20+ years of experience – Variety of industries represented with an emphasis on Aerospace and Defense Historical Data Collection – Data were collected from 25 projects from the Aerospace and Defense application domain – Collected COSYSMO size, effort, and cost driver data – Collected volatility data: added, modified, and deleted requirements over time 14 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Expert Judgment Systems Engineering Effort Penalty Due to Volatility Life Cycle Weighting Factors (n = 27) 11.0 10.0 9.0 8.0 7.0 6.0 5.0 4.0 3.0 2.0 1.0 0.0 Conceptualize Development Operational Test & Evaluation Transition to Operation Data collected from two workshops: 25th Annual USC CSSE COCOMO and the 2011 USC-CSSE ARR 15 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Requirements Volatility Rating Levels Characteristic Rating Very Low Low Moderate High Very High Fully Mostly Generally Somewhat No agreement Level of uncertainty in key customer requirements, mission objectives, and stakeholder needs Very Low Low Moderate High Very High Number of co-dependent systems with influence on system requirements Very Low Low Moderate High Very High Strength of you organization’s requirements development process and level of change control rigor Very High High Moderate Low Very Low Precedentedness of the system , use of mature technology Very High High Moderate Low Very Low Stability of the customer’s organization and business environment Very High High Moderate Low Very Low Experience level of the systems engineering team in requirements analysis and development Very High High Moderate Low Very Low System requirements baselined and agreed to by key stakeholders Developed based on surveys of experienced S/W and Systems Engineers (N =38) 16 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Requirements Volatility Ratings % volatility per life cycle phase % Volatility per Life Cycle Phase 30% 25% 20% mean + 2σ Very Low mean + 1σ Low 15% 10% 5% Moderate High mean Very High mean - 1σ 0% Conceptualize Development Operational Test & Eval Transition to Operation Based on expert judgment collected through three workshops (N = 36) and historical project data (N = 25) 17 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Regression and Bayesian Calibration • COSYSMO can be described as multiple regression model • The model is linearized using a logarithmic transformation ln( SEhrs) ln A 1.06 ln( SIZE ) 1 SF ln( SIZE ) ln( EM ) • The data-determined coefficient β1 is combined with the a-priori expert judgment to develop the Bayesian calibrated model A posteriori Bayesian Update 1.96 2.09 2.01 Data Analysis A priori Expert Judgment University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Model Performance Comparison • Model performance evaluated using the baseline model and a model with local calibration • The performance of COSYSMO improves by including the requirements volatility factor Model Academic COSYMO Req. Volatility Model PRED(.15 48% ) 72% Estimation Accuracy PRED(.20 PRED(.30 52% 80% ) ) 84% 88% MMRE 21% 15% Calibrated Model* Academic COSYMO* 52% 64% 84% 20% Req. Volatility Model* 76% 84% 88% 15% 19 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Coefficient of Determination (n=25) Academic COSYSMO Estimated Systems Engineering Size (with diseconomies of scale) COSYSMO with Requirements Volatility Factor Estimated Systems Engineering Size (with diseconomies of scale) * Due to proprietary reasons only the analysis of the model accuracy is shown, not the data itself 20 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Sensitivity Analysis • Scenarios 1 and 2 • Standard deviation in the requirements volatility weighting factor (wl) • Scenarios 3 and 4 • Combination of cases: • Scenario 1-3, 1-4 • Scenario 2-3, 2-4 Source: Guidelines for the Economic Analysis of Projects (1997) Scenario 1 % Volatility per Life Cycle Phase • Standard deviation in the volatility life cycle profile (Θl ) Conceptualize % Volatility per Life Cycle Phase • Evaluated the sensitivity of the model’s results to the variability of key parameters Development Operational Test & Eval Scenario 2 Conceptualize Development Operational Test & Eval Transition to Operations Transition to Operations 21 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Cross-Validation Results • The 25 projects were randomly divided into K=6 subsets • One of the subsets is excluded from the data set from which the model is built • The resulting model is used to predict effort for the excluded cases Estimation Accuracy Calibrated Model PRED(.15) PRED(.20) PRED(.30) MMRE Academic COSYMO 52% 64% 84% 20% Req. Volatility Model 76% 84% 88% 15% Cross-Validation 76% 80% 88% 16% Cross-Val Scenario 1 72% 80% 88% 16% Cross-Val Scenario 2 72% 80% 88% 16% Cross-Val Scenario 3 76% 80% 88% 16% Cross-Val Scenario 4 68% 80% 88% 16% Cross-Val Scenario 1-3 72% 76% 88% 17% Cross-Val Scenario 1-4 76% 80% 88% 16% Cross-Val Scenario 2-3 76% 80% 88% 16% Cross-Val Scenario 2-4 64% 84% 88% 17% Improvement holds across the scenarios 22 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Conclusions • Observations from the literature and workshop surveys were used to develop a mathematical framework for quantifying the impact of requirements volatility on systems engineering effort • An evaluation of the model was performed by comparing its prediction accuracy against Academic COSYSMO • The results indicate an improvement in effort prediction accuracy and MMRE • Cross validation and sensitivity analysis were performed to demonstrate the accuracy of the model in predicting effort for new projects 23 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Future Work • Obtain additional project data from other organizations – The external validity of this study will be limited by the engineering organizations that contribute data and the background of the industry experts that participate in the research • Evaluate the effect of reuse on the results and its potential interaction with requirements volatility • Further work is needed to complete the characterization of the model performance depending on the type of change: added, modified or deleted 24 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review References • B. Boehm and W. Royce. Ada COCOMO and the Ada Process Model, TRW Defense Systems Group, 1989 • B. Boehm, Software risk management: Principles and practices, IEEE Software 1 (1991), 32-41. • B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E., Horowitz, R. Madachy, D.J. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, New York, NY, 2000 • Economics and Development Resource Center .1997. Guidelines for the Economic Analysis of Projects. • S. Conte, H. Dunsmore, and V. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company, 1986. • General Accounting Office, "Stronger management practices are needed to improve DoD’s software-intensive weapon acquisitions (GAO-04-393)," 2004. • ISO/IEC, "ISO/IEC 15288:2002 (e) systems engineering - system life cycle processes," 2002. • G. Kotonya and I. Sommerville, Requirements engineering: Processes and techniques, John Wiley & Sons, New York, NY, 1998. • MIL-STD-498, "Software development and documentation," 1994. • D. Reifer, Requirements management: The search for nirvana, IEEE Software 17(3) (2000), 45-47 • D. Rhodes, R. Valerdi and G. Roedler, Systems engineering leading indicators for assessing program and technical effectiveness, Systems Engineering 12(1) (2009), 21-35. • Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department. • Wang, G., Boehm, B., Valerdi, R., and Shernoff, A. (2008). “Proposed Modification to COSYSMO Estimating Relationship.” Technical Report. University of Southern California, Center for Systems and Software Engineering. 25 University of Southern California Center for Systems and Software Engineering USC CSSE 2012 Annual Research Review Back-up 26 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Evaluation of Model Performance • The cost estimation accuracy of COSYSMO was compared to the accuracy of the model with the requirements volatility factor • The model was evaluated using predictive accuracy levels, Mean Magnitude of Relative Error (MMRE) and the coefficient of determination (R2) • The prediction accuracy level is defined as: k PRED (l ) n Where k = number of projects in the set whose Magnitude of Relative Error is ≤ l Source: Conte et al., (1986) 27 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Model Comparison Test (F-ratio) • The F-test is typically used to compare regression models and determine whether the null hypothesis is supported or refuted • In this case, the null hypothesis is that the simpler model (without a requirements volatility factor) is correct ( RSS NH RSS AH ) /( df NH df AH ) F MSE AH • Values of the F ratio near one (1) support the null hypothesis, while larger values favor the alternative hypothesis • The F-value was calculated to be 7.94 with > 95% confidence level: Supports the alternative hypothesis 28 University of Southern California Center for Systems and Software Engineering USC-CSSE 2012 Annual Research Review Organizations that Participated in the Research • The Aerospace Corporation • Lockheed Martin • Northrop Grumman Corporation • Ericsson España • The Boeing Company • Rolls Royce • Raytheon • Softstar • United Launch Alliance • Texas Tech • BAE • The US Army • TI Metricas Ltda. • The US Navy • IBM • The US Air force • Distributed Management • The Australian Department of Defense • MIT • Samsung SDS • USC 29