University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Mathematical Formulation and
Validation of the Impact of
Requirements Volatility on
Systems Engineering Effort
November 2nd, 2011
Mauricio E. Peña
Dr. Ricardo Valerdi
1
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Outline
• Motivation and introduction
• Implications to COSYSMO
• Research methods
• Observations from exploratory research
• Proposed model of the impact of requirements
volatility on Systems Engineering effort
• Preliminary model validation
• Conclusions and next steps
2
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Importance of Understanding
Requirements Volatility
• Requirements volatility has been identified by numerous
research studies as a risk factor and cost-driver of systems
engineering projects1
• Requirements changes are costly, particularly in the later
stages of the lifecycle process because the change may require
rework of the design, verification and deployment plans2
• The Government Accountability Office (GAO) concluded in a
2004 report on the DoD’s acquisition of software-intensive
weapons systems that missing, vague, or changing
requirements are a major cause of project failure3
System developers often lack effective methods and tools
to account for and manage requirements volatility
Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393
3
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Requirements Volatility is Expected
• Changes to requirements are a part of our
increasingly complex systems & dynamic business
environment
– Stakeholders needs evolve rapidly
– The customer may not be able to fully specify the system
requirements up front
– New requirements may emerge as knowledge of the system
evolves
– Requirements often change during the early phases of the
project as a result of trades and negotiations
Requirements volatility must be anticipated and managed
Sources: Kotonya and Sommerville (1995); Reifer (2000)
4
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Requirements Volatility Definitions
• Requirements volatility is typically defined as the
change in requirements (added, deleted, and
modified) over a given time interval
• Also known as:
• Requirements creep: An increase in scope and
number of system requirements
• Requirements churn: Instability in the requirements
set – requirements are modified or re-worked without
necessarily resulting in an increase in the total
number of requirements
Source: MIL-STD-498 (1994)
5
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
CSSE Parametric Cost Models
• The Constructive Systems Engineering Cost Model
(COSYSMO) was developed by the USC Center for
Software and Systems Engineering (CSSE) in
collaboration with INCOSE and Industry affiliates
• COSYSMO is the first generally-available parametric
cost model designed to estimate Systems
Engineering effort
• Built on experience from COCOMO 1981, COCOMO II
• During the development of COSYSMO, volatility was
identified as a relevant adjustment factor to the
model’s size drivers
Source: 7th Annual Practical Software and Systems Measurement Conference. COSYSMO Workshop, Boehm
6
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
COSYSMO 2.0 Operational Concept
Proposed
Volatility Factor
Source: Fortune (2009)
7
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Life Cycle Phase Definition
• Conceptualize phase focuses on identifying stakeholder needs,
exploring different solution concepts, and proposing candidate
solutions.
• The Development phase involves refining the system
requirements, creating a solution description, and building a
system.
• The Operational Test & Evaluation Phase involves
verifying/validating the system and performing the appropriate
inspections before it is delivered to the user
• The Transition to Operation Phase involves the transition to
utilization of the system to satisfy the users’ needs.
Source: Valerdi (2008); ISO/IEC 15288 (2002)
8
University of Southern California
26th Annual COCOMO Forum
Center for Systems and Software Engineering
Research Methods
USC
University of Southern California
C S E
Center for Software Engineering
Analyze Existing
literature
Perform
Behavioral Analysis
1
Literature
Review & 6
Workshops
completed to
date
2
In Progress
Identify Relative
Significance
3
A-PRIORI MODEL
+
SAMPLING DATA
=
A-POSTERIORI MODEL
Perform ExpertJudgement, Delphi
Assessment
4
Gather Project Data
5
Determine Bayesian
A-Posteriori Update
6
Gather more data;
refine model
7
USC-CSE Annual Research Review – 3/17/03
Source: Boehm et al (2000)
8
9
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Observations from the Literature and
Exploratory Research
1. Requirements volatility is caused by an identifiable
set of project and organizational factors.
2. The level of requirements volatility is a function of
the system life cycle phase.
3. Requirements volatility leads to an increase in
project size and cost
4. The cost and effort impact of a requirements
change increases the later the change occurs in the
system life cycle
5. The impact of requirements volatility varies
depending on the type of change: added, deleted, or
modified
10
University of Southern California
26th Annual COCOMO Forum
Center for Systems and Software Engineering
Systems Engineering Effort Penalty Due to Volatility
Life Cycle Effort Penalty due to Volatility
(n = 27)
11.0
10.0
9.0
8.0
7.0
6.0
5.0
4.0
3.0
2.0
1.0
0.0
Conceptualize
Development
Operational Test
& Evaluation
Transition to
Operation
Data collected from two workshops: 25th Annual USC CSSE COCOMO and the 2011 USC-CSSE Annual Research Review
11
University of Southern California
26th Annual COCOMO Forum
Center for Systems and Software Engineering
Estimated Life Cycle Effort Penalty per
Change Category (n = 9)
Estimated SE Effort Penalty due to Volatility
10.5
added
deleted
9.0
modified
7.5
6.0
4.5
3.0
1.5
0.0
-1.5
Conceptualize
Development
Operational Test
& Evaluation
Transition to
Operation
Results from a two-round Delphi Survey held at the 2011 Practical Software and Systems Measurement Conference
12
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Mathematical Formulation of
Requirements Volatility ( 1 of 2)
  REVL

Req  1  
 wv    R0 

  100
Where,
R0 = Baseline number of requirements
Req = Equivalent number of requirements
wv = Requirements Volatility weighting factor
• The effective increase in the number of requirements would
result in an associated increase in systems engineering effort
Sources: Boehm, B., et al (2000)
13
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Mathematical Formulation of
Requirements Volatility ( 2 of 2)
wv   wa ,l a ,l  wd ,l d ,l  wm,l m,l 
l
Where
wv = Requirements volatility weighting factor
wx,l = Weighting factor for added, deleted, or
modified requirements
Θx,l = % of total requirements changes that were
added, deleted or modified
l = lifecycle phases
Pena (2010)
14
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Preliminary Model Validation
•
Data were collected from nine projects from a space
systems application domain
– Number of requirements, interfaces, algorithms operational
scenarios (estimated systems engineering size)
– Added, modified, and deleted requirements over time
•
The cost estimation accuracy of COSYSMO was
compared to the accuracy of the model with the
requirements volatility size driver adjustment factor
•
The models were tested using the coefficient of
determination (R2) and predictive accuracy levels
•
The relationship between requirements volatility
and systems engineering size was also examined
15
University of Southern California
26th Annual COCOMO Forum
Center for Systems and Software Engineering
Actual Systems Engineering Effort
(Person Months)
Baseline Coefficient of Determination
(n = 9)
R2 = 0.78
Estimated Systems Engineering Size (Eqv. Requirements)
* Due to proprietary reasons only the analysis of the model accuracy is shown, not the data itself
16
University of Southern California
26th Annual COCOMO Forum
Center for Systems and Software Engineering
Actual Systems Engineering Effort
(Person Months)
Coefficient of Determination with
Volatility Factor (n = 9)
R2 = 0.85
Estimated Systems Engineering Size (Eqv. Requirements)
17
PRED (l ) 
k
n
University of Southern California
26th Annual COCOMO Forum
Center for Systems and Software Engineering
Prediction Accuracy Level
The prediction accuracy at a particular level (l) is defined as
PRED (l ) 
k
n
Where,
n = set of projects
k = number of projects in the set whose Magnitude of Relative
Error is ≤ l
Prediction Accuracy (l)
Estimates with COSYSMO
Estimate with REVL
PRED(20)
11%
33%
PRED(30)
44%
56%
Source: Conte, Dunsmore, and Shen (1986)
18
University of Southern California
26th Annual COCOMO Forum
Center for Systems and Software Engineering
Requirements Volatility (%)
Relationship Between Requirements
Volatility and Sys. Engr. Size
R2 = 0.29
Estimated Systems Engineering Size (Eqv. Requirements)
19
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Requirements Volatility Factor
Relationship Between the Volatility
Adjustment Factor and Sys. Engr. Size
R2 = 0.34
Estimated Systems Engineering Size (Eqv. Requirements)
20
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
Conclusions – Next Steps
• Observations from the literature and workshops were used to
develop a mathematical framework for quantifying the impact of
requirements volatility on systems engineering effort
• A preliminary validation of the model was performed by
comparing its prediction accuracy against COSYSMO
• The preliminary results indicate an improvement in systems
engineering effort prediction accuracy when a requirements
volatility factor is added to the model
• The correlation between the requirements volatility factor and
the estimated systems engineering size was examined – the
results were not conclusive
• Additional project data will be collected to complete the model
validation
21
University of Southern California
Center for Systems and Software Engineering
26th Annual COCOMO Forum
References
• B. Blanchard and W. Fabrycky, Systems engineering & analysis, Prentice Hall, New York, NY, 1998.
• B. Boehm, Software risk management: Principles and practices, IEEE Software 1 (1991), 32-41.
• B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E., Horowitz, R. Madachy, D.J. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, New York,
NY, 2000
• S. Conte, H. Dunsmore, and V. Shen. Software Engineering Metrics and Models. Benjamin/Cummings Publishing Company, 1986.
• Department of Defense, “Instruction 5000.02. Operation of the Defense Acquisition System,” 2008.
• S. Ferreira, J. Collofello, D. Shunk and G. Mackulak, Understanding the Effects of Requirements Volatility in Software Engineering by Using Analytical Modeling and Software
Process Simulation, Journal of Systems and Software 82 (2009) 1568-1577.
• General Accounting Office, "Stronger management practices are needed to improve DoD’s software-intensive weapon acquisitions (GAO-04-393)," 2004.
• D. Houston, "A software project simulation model for risk management," Ph.D. Dissertation, Arizona State University, 2000.
• ISO/IEC, "ISO/IEC 15288:2002 (e) systems engineering - system life cycle processes," 2002.
• C. Jones, Assessment and Control of Software Risks, Prentice Hall, Inc. Englewood Cliffs, New Jersey, 1994.
• C. Jones, Strategies for managing requirements creep, IEEE Computer, Vol 20 (1996), pp. 92-94
• G. Kotonya and I. Sommerville, Requirements engineering: Processes and techniques, John Wiley & Sons, New York, NY, 1998.
• G.P. Kulk, and C. Verhoef, Quantifying Requirements Volatility Effects, Science of Computer Programming, 72, 136-175, 2008.
• Y. Malaiya, and J. Denton, Requirements Volatility and Defect Density, Proceedings of the International Symposium on Software Reliability Engineering, 1999
• MIL-STD-498, "Software development and documentation," 1994.
• S. Nidumolu, Standardization, Requirements Uncertainty and Software Project Performance, Information and Management. Vol 31 (No. 3) (1996), pp 135-150
• M. Pena, Characterizing the Impacts of Requirements Volatiliy, Proceedings of the 25th Annual COCOMO Forum, 2010
• D. Pfahl, and K. Lebsanft, Using Simulation to Analyze the Impact of Software Requirements Volatility on Project Performance, Information and Software Technology, Vol 42 (No.
14) (2000), pp 1001-1008.
• D. Reifer, Requirements management: The search for nirvana, IEEE Software 17(3) (2000), 45-47
• D. Rhodes, R. Valerdi and G. Roedler, Systems engineering leading indicators for assessing program and technical effectiveness, Systems Engineering 12(1) (2009), 21-35.
• R. Valerdi, The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems, Saarbrücken, Germany,
VDM Verlag, 2008.
• D. Zowghi and N. Nurmuliani, A study of the impact of requirements volatility on software project performance, Proceedings of the Ninth Asia-Pacific Software Engineering
Conference, 2002.
22