University of Southern California
Center for Systems and Software Engineering
USC CSSE 2012 Annual Research Review
Mathematical Formulation and
Validation of the Impact of
Requirements Volatility on
Systems Engineering Effort
March 6, 2012
Mauricio E. Peña
1
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Outline
• Motivation and introduction
• Research methods
• Observations from the research
• Model Development
• Results
– Model Calibration
– Evaluation of Model Performance
– Sensitivity Analysis
– Cross-Validation
• Conclusion
2
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Importance of Understanding
Requirements Volatility
• Requirements volatility has been identified by numerous
research studies as a risk factor and cost-driver of systems
engineering projects1
• Requirements changes are costly, particularly in the later
stages of the lifecycle process because the change may require
rework of the design, verification and deployment plans2
• The Government Accountability Office (GAO) concluded in a
2004 report on the DoD’s acquisition of software-intensive
weapons systems that missing, vague, or changing
requirements are a major cause of project failure3
System developers often lack effective methods and tools
to account for and manage requirements volatility
Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393
3
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Requirements Volatility is Expected
• Changes to requirements are a part of our
increasingly complex systems & dynamic business
environment
– Stakeholders needs evolve rapidly
– The customer may not be able to fully specify the system
requirements up front
– New requirements may emerge as knowledge of the system
evolves
– Requirements often change during the early phases of the
project as a result of trades and negotiations
Requirements volatility must be anticipated and managed
Sources: Kotonya and Sommerville (1995); Reifer (2000)
4
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
CSSE Parametric Cost Models
• The Constructive Systems Engineering Cost Model
(COSYSMO) was developed by the USC Center for
Software and Systems Engineering (CSSE) in
collaboration with INCOSE and Industry affiliates
• COSYSMO is the first generally-available parametric
cost model designed to estimate Systems
Engineering effort
• Built on experience from COCOMO 1981, COCOMO II
• During the development of COSYSMO, volatility was
identified as a relevant adjustment factor to the
model’s size drivers
Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm
5
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
COSYSMO Operational Concept
Volatility Factor
# Requirements
# Interfaces
# Scenarios
# Algorithms
Size
Drivers
Effort
Multipliers
- Application factors
-8 factors
- Team factors
-6 factors
Source: Valerdi (2005)
COSYSMO
Effort
Calibration
6
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Research Methods
USC
University of Southern California
C S E
Center for Software Engineering
Analyze Existing
literature
1
Literature
Review & 6
Workshops
completed
Data gathered
from 25
industry
projects
Perform
Behavioral Analysis
2
Identify Relative
Significance
3
A-PRIORI MODEL
+
SAMPLING DATA
=
A-POSTERIORI MODEL
Perform ExpertJudgement, Delphi
Assessment
4
Gather Project Data
We are here
5
Determine Bayesian
A-Posteriori Update
6
Gather more data;
refine model
7
USC-CSE Annual Research Review – 3/17/03
Source: Boehm et al (2000)
8
7
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Organizations that Participated in
the Research
• The Aerospace Corporation
• Lockheed Martin
• Northrop Grumman
Corporation
• Ericsson España
• The Boeing Company
• Rolls Royce
• Raytheon
• Softstar
• United Launch Alliance
• Texas Tech
• BAE
• The US Army
• TI Metricas Ltda.
• The US Navy
• IBM
• The US Air force
• Distributed Management
• The Australian Department of
Defense
• MIT
• Samsung SDS
• USC
8
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Requirements Volatility Observations
1. Requirements volatility is caused by an identifiable
set of project and organizational factors
2. The level of requirements volatility is a function of
the system life cycle phase
3. Requirements volatility leads to an increase in
project size and cost
4. The cost and effort impact of a requirements
change increases the later the change occurs in the
system life cycle
5. The impact of requirements volatility may vary
depending on the type of change: added, deleted, or
modified
9
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Model Form
• Incorporated volatility effects through a scale factor
(SF) added to the diseconomies of scale exponent (E)
n
PM
E  SF


 A  SIZE
  ( EM i )
i 1
• Similar approach used to model volatility effects in
Ada COCOMO1
• Prior research points to the compounding or
exponential effect of project factors with variable life
cycle impact2
Source: 1: Boehm, B. and Royce, W. (1989); 2: Wang, G. et al., (2008)
10
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Volatility Scale Factor

 REVL

SFw   0.05  
 wvl  
 100


Expected REVL is
rated as Very Low,
Low, Moderate, High
and Very High
• Where
– REVL = The % of the baseline requirements that is expected to change over
the system lifecycle
– wvl = aggregate lifecycle phase volatility weighting factor
wv1   wl l 
• And:
l
– wl = weighting factor for each life cycle phase1
– Θl = % of total requirements changes per life cycle phase
1Life
Cycle Phases: Conceptualize, Development, Operational Test and Evaluation, and Transition to Operation
11
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Data Collection
•
•
Collected expert judgment on REVL and weighting
factors through surveys and workshops
–
Software and Systems Engineers with 20+ years of
experience
–
Variety of industries represented with an emphasis on
Aerospace and Defense
Historical Data Collection
–
Data were collected from 25 projects from the Aerospace
and Defense application domain
–
Collected COSYSMO size, effort, and cost driver data
–
Collected volatility data: added, modified, and deleted
requirements over time
12
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Model Calibration
A Priori Model
Initial parameter
mean and variance
Historical data
Data-determined Model
Regression Analysis; OLS
Mean and Variance
Expert-judgment estimates
A Posteriori Model
Bayesian Analysis
Formally combines expert judgment
With sample data
Source: Boehm et al. (2000); Nguyen (2010)
Updated
parameter mean
and variance
Optimized combination
of data sources to
increase precision
13
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Requirements Volatility (REVL)
Rating Levels
Characteristic
Rating
Very Low
Low
Moderate
High
Very High
Fully
Mostly
Generally
Somewhat
No
agreement
Level of uncertainty in key customer
requirements, mission objectives,
and stakeholder needs
Very Low
Low
Moderate
High
Very High
Number of co-dependent systems
with influence on system
requirements
Very Low
Low
Moderate
High
Very High
Strength of you organization’s
requirements development process
and level of change control rigor
Very High
High
Moderate
Low
Very Low
Precedentedness of the system , use
of mature technology
Very High
High
Moderate
Low
Very Low
Stability of the customer’s
organization and business
environment
Very High
High
Moderate
Low
Very Low
Experience level of the systems
engineering team in requirements
analysis and development
Very High
High
Moderate
Low
Very Low
System requirements baselined and
agreed to by key stakeholders
Developed based on surveys of experienced S/W and Systems Engineers (N =38)
14
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Requirements Volatility Profile
% volatility per life cycle phase
% Volatility per Life Cycle Phase
30%
25%
20%
mean + 2σ
Very Low
mean + 1σ
Low
15%
10%
5%
Moderate
High
mean
Very High
mean - 1σ
0%
Conceptualize Development
Operational
Test & Eval
Transition to
Operation
Based on expert judgment collected through three workshops (N = 36) and historical project data (N = 25)
15
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Regression and Bayesian Calibration
• COSYSMO can be described as multiple regression model
• The model is linearized using a logarithmic transformation
ln( SEhrs)  ln A  1.06  ln( SIZE )  1  SF ln( SIZE )  ln( EM )
• The data-determined coefficient β1 is combined with the a-priori
expert judgment to develop the Bayesian calibrated model
A posteriori
Bayesian Update
1.96
2.09
2.01
Data Analysis
A priori
Expert Judgment
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Model Performance Comparison
• Model performance evaluated using the baseline
model and a model with local calibration
• The performance of COSYSMO improves by
including the requirements volatility factor
Model
Estimation Accuracy
PRED(.15)
PRED(.20)
PRED(.30)
MMRE
Academic COSYMO
48%
52%
80%
21%
Req. Volatility Model
72%
80%
88%
16%
Academic COSYMO*
52%
64%
84%
20%
Req. Volatility Model*
76%
84%
88%
15%
Calibrated Model*
17
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Coefficient of Determination (n=25)
Academic COSYSMO
Estimated Systems Engineering Size (with diseconomies of scale)
COSYSMO with Requirements
Volatility Factor
Estimated Systems Engineering Size (with diseconomies of scale)
* Due to proprietary reasons only the analysis of the model accuracy is shown, not the data itself
18
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Sensitivity Analysis
• Scenarios 1 and 2
• Standard deviation in the
requirements volatility
weighting factor (wl)
• Scenarios 3 and 4
• Combination of cases:
• Scenario 1-3, 1-4
• Scenario 2-3, 2-4
Source: Guidelines for the Economic Analysis of Projects (1997)
Scenario 1
% Volatility per Life Cycle Phase
• Standard deviation in the
volatility life cycle profile (Θl )
Conceptualize
% Volatility per Life Cycle Phase
• Evaluated the sensitivity of
the model’s results to the
variability of key parameters
Development
Operational
Test & Eval
Scenario 2
Conceptualize
Development
Operational
Test & Eval
Transition to
Operations
Transition to
Operations
19
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Cross-Validation Results
• The 25 projects were randomly divided into K=6 subsets
• One of the subsets is excluded from the data set from which the model
is built
• The resulting model is used to predict effort for the excluded cases
Estimation Accuracy
Calibrated Model
PRED(.15)
PRED(.20)
PRED(.30)
MMRE
Academic COSYMO
52%
64%
84%
20%
Req. Volatility Model
76%
84%
88%
15%
Cross-Validation
76%
80%
88%
16%
Cross-Val Scenario 1
72%
80%
88%
16%
Cross-Val Scenario 2
72%
80%
88%
16%
Cross-Val Scenario 3
76%
80%
88%
16%
Cross-Val Scenario 4
68%
80%
88%
16%
Cross-Val Scenario 1-3
72%
76%
88%
17%
Cross-Val Scenario 1-4
76%
80%
88%
16%
Cross-Val Scenario 2-3
76%
80%
88%
16%
Cross-Val Scenario 2-4
64%
84%
88%
17%
Improvement
holds across
the scenarios
20
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Conclusions
• Observations from the literature and workshop
surveys were used to develop a mathematical
framework for quantifying the impact of requirements
volatility on systems engineering effort
• An evaluation of the model was performed by
comparing its prediction accuracy against Academic
COSYSMO
• The results indicate an improvement in effort
prediction accuracy and MMRE
• Cross validation and sensitivity analysis were
performed to demonstrate the accuracy of the model
in predicting effort for new projects
21
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Future Work
• Obtain additional project data from other
organizations
– The external validity of this study will be limited by the
engineering organizations that contribute data and the
background of the industry experts that participate in the
research
• Evaluate the effect of reuse on the results and its
potential interaction with requirements volatility
• Further work is needed to complete the
characterization of the model performance
depending on the type of change: added, modified or
deleted
22
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
References
• B. Boehm and W. Royce. Ada COCOMO and the Ada Process Model, TRW Defense Systems Group, 1989
• B. Boehm, Software risk management: Principles and practices, IEEE Software 1 (1991), 32-41.
• B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E., Horowitz, R. Madachy, D.J. Reifer, and B. Steece, Software
Cost Estimation with COCOMO II, Prentice Hall, New York, NY, 2000
• Economics and Development Resource Center .1997. Guidelines for the Economic Analysis of Projects.
• S. Conte, H. Dunsmore, and V. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing
Company, 1986.
• General Accounting Office, "Stronger management practices are needed to improve DoD’s software-intensive weapon
acquisitions (GAO-04-393)," 2004.
• ISO/IEC, "ISO/IEC 15288:2002 (e) systems engineering - system life cycle processes," 2002.
• G. Kotonya and I. Sommerville, Requirements engineering: Processes and techniques, John Wiley & Sons, New York,
NY, 1998.
• MIL-STD-498, "Software development and documentation," 1994.
• D. Reifer, Requirements management: The search for nirvana, IEEE Software 17(3) (2000), 45-47
• D. Rhodes, R. Valerdi and G. Roedler, Systems engineering leading indicators for assessing program and technical
effectiveness, Systems Engineering 12(1) (2009), 21-35.
• Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of
Southern California, Industrial and Systems Engineering Department.
• Wang, G., Boehm, B., Valerdi, R., and Shernoff, A. (2008). “Proposed Modification to COSYSMO Estimating
Relationship.” Technical Report. University of Southern California, Center for Systems and Software Engineering.
23
University of Southern California
Center for Systems and Software Engineering
USC CSSE 2012 Annual Research Review
Back-up
24
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
COSYSMO Cost Estimating
Relationships (CERs)
n
SE _ Hrs  A  SIZE    ( EM i )
E
i 1
SE_Hrs = Systems Engineering effort (hours)
A = calibration constant derived from historical project data
SIZE = measure of functional size of the system (number of
requirements, interfaces, algorithms, operational scenarios)
n = number of cost drivers (14)
EM = effort multiplier for the ith cost driver
The exponent (E) accounts for diseconomies of scale
Source: Valerdi (2005)
25
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Expert Judgment
Systems Engineering Effort Penalty Due to Volatility
Life Cycle Weighting Factors (n = 27)
11.0
10.0
9.0
8.0
7.0
6.0
5.0
4.0
3.0
2.0
1.0
0.0
Conceptualize
Development
Operational Test
& Evaluation
Transition to
Operation
Data collected from two workshops: 25th Annual USC CSSE COCOMO and the 2011 USC-CSSE ARR
26
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Evaluation of Model Performance
•
The cost estimation accuracy of COSYSMO was
compared to the accuracy of the model with the
requirements volatility factor
•
The model was evaluated using predictive accuracy
levels, Mean Magnitude of Relative Error (MMRE)
and the coefficient of determination (R2)
•
The prediction accuracy level is defined as:
k
PRED (l ) 
n
Where
k = number of projects in the set whose Magnitude of
Relative Error is ≤ l
Source: Conte et al., (1986)
27
University of Southern California
Center for Systems and Software Engineering
USC-CSSE 2012 Annual Research Review
Model Comparison Test (F-ratio)
• The F-test is typically used to compare regression models and
determine whether the null hypothesis is supported or refuted
•
In this case, the null hypothesis is that the simpler model
(without a requirements volatility factor) is correct
( RSS NH  RSS AH ) /( df NH  df AH )
F
MSE AH
• Values of the F ratio near one (1) support the null hypothesis,
while larger values favor the alternative hypothesis
• The F-value was calculated to be 7.62 with > 95% confidence
level: Supports the alternative hypothesis
28