University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Quantifying the Impact of
Requirements Volatility on
Systems Engineering Effort
October 18, 2012
Mauricio E. Peña
Ricardo Valerdi
1
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Outline
• Motivation and Introduction
• Research Methods
• Model Development
• Results
• Cross-Validation and Sensitivity Analysis
• Conclusion
• Threats to Validity
• Future Research
2
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Importance of Understanding
Requirements Volatility
• Requirements volatility has been identified by numerous
research studies as a risk factor and cost-driver of systems
engineering projects1
• Requirements changes are costly, particularly in the later
stages of the lifecycle process because the change may require
rework of the design, verification and deployment plans2
• The Government Accountability Office (GAO) concluded in a
2004 report on the DoD’s acquisition of software-intensive
weapons systems that missing, vague, or changing
requirements are a major cause of project failure3
System developers often lack effective methods and tools
to account for and manage requirements volatility
Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393
3
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Requirements Volatility is Expected
• Changes to requirements are a part of our
increasingly complex systems & dynamic business
environment
– Stakeholders needs evolve rapidly
– The customer may not be able to fully specify the system
requirements up front
– New requirements may emerge as knowledge of the system
evolves
– Requirements often change during the early phases of the
project as a result of trades and negotiations
Requirements volatility must be anticipated and managed
Sources: Kotonya and Sommerville (1995); Reifer (2000)
4
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
CSSE Parametric Cost Models
• The Constructive Systems Engineering Cost Model
(COSYSMO) was developed by the USC Center for
Software and Systems Engineering (CSSE) in
collaboration with INCOSE and Industry affiliates
• COSYSMO is the first generally-available parametric
cost model designed to estimate Systems
Engineering effort
• Built on experience from COCOMO 1981, COCOMO II
• During the development of COSYSMO, volatility was
identified as a relevant adjustment factor to the
model’s size drivers
Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm
5
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Research Methods
USC
University of Southern California
C S E
Center for Software Engineering
Analyze Existing
literature
1
Literature
Review & 6
Workshops
completed
Data gathered
from 25
industry
projects
Perform
Behavioral Analysis
2
Identify Relative
Significance
3
A-PRIORI MODEL
+
SAMPLING DATA
=
A-POSTERIORI MODEL
Perform ExpertJudgement, Delphi
Assessment
4
Gather Project Data
We are here
5
Determine Bayesian
A-Posteriori Update
6
Gather more data;
refine model
7
USC-CSE Annual Research Review – 3/17/03
Source: Boehm et al (2000)
8
6
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Organizations that Participated in
the Research
• The Aerospace Corporation
• Lockheed Martin
• Northrop Grumman
Corporation
• Ericsson España
• The Boeing Company
• Rolls Royce
• Raytheon
• Softstar
• United Launch Alliance
• Texas Tech
• BAE
• The US Army
• TI Metricas Ltda.
• The US Navy
• IBM
• The US Air force
• Distributed Management
• The Australian Department of
Defense
• MIT
• Samsung SDS
• USC
7
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Observations from the Literature and
Research Workshops
1. Requirements volatility is caused by an identifiable set
of project and organizational factors
2. The level of requirements volatility is a function of the
system life cycle phase
3. Requirements volatility leads to an increase in project
size and cost
4. The cost and effort impact of a requirements change
increases the later the change occurs in the system life
cycle
5. The impact of requirements volatility may vary
depending on the type of change: added, deleted, or
modified
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston
(2000); Zowghi and Nurmuliani (2002); Ferreira (2002); Kulk and Verhoef 2008
8
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
COSYSMO Cost Estimating
Relationships (CERs)
n
SE _ Hrs  A  SIZE   ( EM i )
E
i 1
SE_Hrs = Systems Engineering effort (hours)
A = calibration constant derived from historical project data
SIZE = measure of functional size of the system (number of
requirements, interfaces, algorithms, operational scenarios)
n = number of cost drivers (14)
EM = effort multiplier for the ith cost driver
The exponent (E) accounts for diseconomies of scale
Source: Valerdi (2005)
9
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Model Form
• Incorporated volatility effects through a scale factor
(SF) added to the diseconomies of scale exponent (E)
PM  A  SIZE
E  SF
n
  ( EM i )
i 1
• Similar approach used to model volatility effects in
Ada COCOMO
• Prior research points to the compounding or
exponential effect of requirements volatility
• Linearized form:
ln(SE _ hrs)  ln A  E  ln(SIZE)  1  SF ln(SIZE)  ln(EMi )
Source: Boehm, B. and Royce, W. (1989); Kulk and Verhoef (2008); Wang, G. et al., (2008)
10
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Volatility Scale Factor
  REVL 

SFw   C  
  wvl 
  100 

Expected REVL is
rated as Very Low,
Low, Moderate, High,
and Very High
• Where
– REVL = The % of the baseline requirements that is expected to change over
the system lifecycle
– C = Scale factor constant = 0.05
– wvl = aggregate lifecycle phase volatility weighting factor
wv1  wconcepconcep  wdevdev  wtest test  wtran _ ops tran _ ops 
• And:
– wl = weighting factor for each life cycle phase1
– Θl = % of total requirements changes per life cycle phase
– l = life cycle phases
1Life
Cycle Phases: Conceptualize, Development, Operational Test and Evaluation, and Transition to Operation
11
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Data Collection
•
•
Collected expert judgment on the causes of
volatility, expected REVL, and life cycle weighting
factors through surveys conducted at 6 workshops
–
Software and Systems Engineers with 20+ years of
experience
–
Variety of industries represented with an emphasis on
Aerospace and Defense
Historical Data Collection
–
Data were collected from 25 projects from the Aerospace
and Defense application domain
–
Collected COSYSMO size, effort, and cost driver data
–
Collected volatility data: added, modified, and deleted
requirements over time
12
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Causes of Requirements Volatility (N = 37)
Internal factors: Change in policies,
organizational structure.
Changes in co-dependent systems
Changes in COTS products
Changes in external environment
(political/business climate)
Immature technology
Customer-requested scope change
Inexperienced staff
Lack of SE process maturity
Poor initial understanding of the
system and customer needs
0%
10%
20%
Strongly Disagree
30%
40%
Disagree
50%
60%
Neither agree nor disagree
70%
Agree
80%
90%
100%
Strongly Agree
13
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Requirements Volatility (REVL)
Rating Levels
Very
Low
< 1.5
Low
Moderate
High
>1.5-2.5
>2.5-3.5
>3.5-4.5
System requirements baselined and
agreed to by key stakeholders
Fully
1
Mostly
2
Level of uncertainty in key
customer requirements, mission
objectives, and stakeholder needs
Very
Low
1
Low
2
Moderate
3
Number of co-dependent systems
with influence on system
requirements
Very
Low
1
Low
2
Strength of your organization’s
requirements development process
and level of change control rigor
Very
High
1
Precedentedness of the system , use
of mature technology
Very
High
1
Stability of stakeholders'
organizations (developer,
customer)
Very
High
1
Experience level of the systems
engineering team in requirements
analysis and development
Very
High
1
Characteristic
Rating
Very
High
> 4.5
Weight
26%
High
4
No
Agreement
5
Very High
5
Moderate
3
High
4
Very High
5
16%
High
2
Moderate
3
Low
4
Very Low
5
8%
High
2
Moderate
3
Low
4
Very Low
5
9%
High
2
Moderate
3
Low
4
Very Low
5
High
2
Moderate
3
Low
4
Very Low
5
Generally Somewhat
3
4
22%
14%
6%
Developed based on surveys of experienced S/W and Systems Engineers (N =37)
14
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Requirements Volatility Profile
% volatility per life cycle phase
% Volatility per Life Cycle Phase
30%
25%
20%
mean + 2σ
Very Low
mean + 1σ
Low
15%
10%
5%
Moderate
High
mean
Very High
mean - 1σ
0%
Conceptualize Development
Operational
Test & Eval
Transition to
Operation
Based on expert judgment collected through three workshops (N = 36) and historical project data (N = 25)
15
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Model Calibration
A Priori Model
Initial parameter
mean and variance
Historical data
Data-determined Model
Regression Analysis; OLS
Mean and Variance
Expert-judgment estimates
A Posteriori Model
Bayesian Analysis
Formally combines expert judgment
With sample data
Source: Boehm et al. (2000); Nguyen (2010)
Updated
parameter mean
and variance
Optimized combination
of data sources to
increase precision
16
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Regression and Bayesian Calibration
The coefficient β1 for the requirements volatility scale factor was
calculated through linear regression analysis using data from the
25 projects
Predictor
Coefficient
Standard
Error
t Stat
P-value
1.961
0.851
2.304
0.03
The data-determined coefficient β1 was formally combined with the
a-priori expert judgment to develop the Bayesian calibrated model
A posteriori
Bayesian Update
1.96
2.09
2.01
*
Data Analysis
A priori
Expert Judgment
17
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Model Performance Comparison
• Model performance evaluated using the baseline
model and a model with local calibration
• The performance of COSYSMO improves by
including the requirements volatility factor
Model
Estimation Accuracy
PRED(.15)
PRED(.20)
PRED(.30)
MMRE
Academic COSYMO
48%
52%
80%
21%
Req. Volatility Model
72%
80%
88%
16%
Academic COSYMO*
52%
64%
84%
20%
Req. Volatility Model*
76%
84%
88%
15%
Calibrated Model*
18
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Coefficient of Determination (n=25)
COSYSMO with Requirements
Volatility Factor
Academic COSYSMO
30
25
Actual Systems Engineering Effort*
(Person-hours)
Actual Systems Engineering Effort*
(Person-hours)
30
R² = 0.85
20
15
10
5
0
0
2
4
6
8
10
12
25
R2 = 0.92
20
15
10
5
0
Systems Engineering Size*
0
2
4
6
8
Systems Engineering Size*
10
12
14
(weighted sum of the number of requirements, operational scenarios, interfaces,
and algorithms; adjusted using diseconomy of scale exponent E )
(weighted sum of the number of requirements, operational scenarios, interfaces, and
algorithms; adjusted using diseconomy of scale exponents E and SF)
* Relative Scale
19
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Scale Factor Correlation Matrix
• The correlation between the Requirements Volatility Scale
factor and COSYSMO cost drivers that are potential sources of
volatility was evaluated
Volatility
Scale Factor
(SF)
Volatility Scale Factor (SF)
Requirements
Understanding
Personnel
experience
Technology
Risk
1
Requirements Understanding
0.3878
1
Personnel experience
0.0074
0.4815
1
Technology Risk
0.4462
0.2060
0.0226
1
• The correlations between predictors were below the
correlation threshold of 0.66 used for COCOMO II and
COSYSMO
20
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Cross-Validation Results
• The 25 projects were randomly divided into K=6 subsets
• One of the subsets is excluded from the data set from which the model
is built
• The resulting model is used to predict effort for the excluded cases
• This method is repeated for all subsets and the mean magnitude of
relative errors (MMRE) and prediction accuracy PRED (l) across all K
trials is calculated
Estimation Accuracy
Calibrated Model
PRED(.15)
PRED(.20)
PRED(.30)
MMRE
Academic COSYMO
52%
64%
84%
20%
Req. Volatility Model
76%
84%
88%
15%
Cross-Validation
76%
80%
88%
16%
21
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Sensitivity Analysis
• Scenario 1:
– None of the requirements
changes occur in the
conceptualize phase
• Scenario 2:
– 66% of the changes occur in the
conceptualize phase
Source: Guidelines for the Economic Analysis of Projects (1997)
% Volatility per Life Cycle Phase
• 2 Scenarios were developed
based on the standard
deviation in the volatility life
cycle profile (Θl)
Conceptualize
% Volatility per Life Cycle Phase
• Evaluated the sensitivity of
the model’s results to the
variability of key parameters
Scenario 1
Development
Operational
Test & Eval
Transition to
Operations
Scenario 2
Conceptualize
Development
Operational
Test & Eval
Transition to
Operations
22
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Sensitivity Analysis Results
• The performance of the model scenarios was calculated using
the K-fold cross-validation method
Estimation Accuracy
Calibrated Model
PRED(.15)
PRED(.20)
PRED(.30)
MMRE
Academic COSYMO
52%
64%
84%
20%
Req. Volatility Model
76%
84%
88%
15%
Cross-Val Scenario 1
72%
80%
88%
16%
Cross-Val Scenario 2
72%
80%
88%
16%
• The improvement in performance of the model with the
requirements volatility scale factor over Academic COSYSMO
remains largely unchanged
23
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Conclusions
• The volatility of requirements throughout a system’s
life cycle was found to be statistically significant
predictor (p = 0.03) of systems engineering effort
• The prediction accuracy of Academic COSYSMO
improved by incorporating a requirements volatility
factor into the model
• Cross validation and sensitivity analyses were
performed to demonstrate the robustness of the
model in predicting effort for new projects
24
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Threats to Validity
• Internal validity
– Data collected from an engineering organization is influenced by many
factors and all the relevant variables cannot be fully controlled as they
would be in a true experiment
• External validity
– Limited by the organizations that contribute data and the background of
the industry experts that participate in the research
– While a model calibrated to a single organization’s data can be useful, it is
less than fully definitive for other organizations.
• Reliability
– Potential inaccuracies in effort and requirements metrics due to changes in
business systems and practices over the time span of the projects
– Mitigated though:
• The use of consistent counting rules and involvement from personnel
familiar with the projects
• Holding workshops to ensure survey respondents had a consistent
understanding of the questions
25
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Future Research
• Increase the quantity and diversity of the data –
calibrate the model using data from additional
organizations
• Evaluate the potential interaction between
requirements volatility and reuse
• Further investigation of the impact of systems
engineering effort depending on the type of change:
added, modified, and deleted
26
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
References
•
Boehm, B., Abts, C., Brown, A.W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B. (2000). Software
Cost Estimation with COCOMO II. Prentice Hall
•
Department of Defense (2010). Quadrennial Defense Review Report
•
Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software
engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp
1568-1577.
•
Wang, G., Boehm, B., Valerdi, R., and Shernoff, A. (2008). “Proposed Modification to COSYSMO Estimating Relationship.”
Technical Report. University of Southern California, Center for Systems and Software Engineering.
•
General Accounting Office (2004). Stronger Management Practices are Needed to Improve DOD’s Software-intensive Weapon
Acquisitions (GAO-04-393). Defense Acquisitions.
•
Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of Defense
Software Engineering. Pp 20-25.
•
Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University
•
ISO/IEC (2008). ISO/IEC 15288:2008 (E) Systems Engineering - System Life Cycle Processes.
•
Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd.
•
Malaiya, Y., and Denton, J. (1999). “Requirements Volatility and Defect Density.” Proceedings of the International Symposium on
Software Reliability Engineering.
•
MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense.
•
Nguyen, V. and Boehm, B. (2010). A COCOMO Extension for Software Maintenance. 25th International Forum on COCOMO and
Systems/Software Cost Modeling
•
Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of Southern
California, Industrial and Systems Engineering Department.
•
Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance.
Proceedings of the Ninth Asia-Pacific Software Engineering Conference
27
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
Back-up
28
University of Southern California
Center for Systems and Software Engineering
2012 COCOMO Forum
REVL (% of baseline requirements changes)
Cumulative Volatility
54%
38%
22%
6%
Conceptualize
Development
Operational
Test & Eval
Transition to
Operations
Back
29