A study of the Causes of Requirements
Volatility and its Impact on Systems
Engineering Effort
COSYSMO Workshop
Center for Software and Systems Engineering, Annual Research
Review
March 11, 2010
Mauricio E. Peña – Ph.D. Student, USC Industrial and Systems
Engineering Department
Motivation
“Requirements are the foundation of the project. They
form the basis for design, manufacture, test and
operations….changes in requirements later in the
development cycle can have a significant cost impact,
possibly resulting in cancellation.”
INCOSE Systems Engineering Handbook (2006)
2
Objectives
• Provide and overview of requirements volatility
and its importance to engineering projects
• Introduce the research questions & research plan
• Discuss the state of the art and implications to
COSYSMO
• Solicit feedback:
– On a requirements volatility causal model
– On a requirements volatility survey
3
Requirements Volatility Overview
Requirements volatility is the change
in requirements (added, deleted, and
modified) over a given time interval
Number of Requirements Over Time
1600
1200
[MIL-STD-498, 1994, Ferreira, 2002]
800
Also known as:
400
Requirements creep: An increase in
scope and number of system
requirements
Requirements churn: Instability in the
requirements set – requirements are
modified or re-worked without
necessarily resulting in an increase in
the total number of requirements
CDR
0
Dec-09
Mar-10
Jun-10
Sep-10
Jan-11
Change in Requirements Over Time
500
New
Modified
Deleted
400
300
200
100
0
J
F M A M
[Hammer et al, 1998]
J
J
A S O N
4
Importance of Understanding
Requirements Volatility
• Requirements volatility has been identified by numerous research
studies as a risk factor and cost-driver of systems engineering
projects [Ferreira 2002, Boehm 1991]
• Requirements changes are costly, particularly in the later stages of
the lifecycle process because the change may require rework of the
design, verification and deployment plans [Kotonya and Sommerville, 1995]
• Requirements volatility trends are considered leading indicators of
project performance as defined by the Lean Advancement Initiative
Systems Engineering Leading Indicators Guide (2007)
• The Government Accountability Office (GAO) concluded in a 2004
report on the DoD’s acquisition of software-intensive weapons
systems that missing, vague, or changing requirements are a
major cause of project failure
5
Research Questions
• Research Question 1: What causes
requirements volatility?
• Research Question 2: How does requirements
volatility impact systems engineering effort?
6
Research Plan
Completed
Literature
In-progress
Planned
Review
Tested
Hypothesis
Observations
Pilot RV survey
Statistical
Analysis
Causal Model
COSYSMO Workshop
Discussion
Updated Causal
Model (hypothesis)
Industry Data
Delphi
Survey
Delphi
Survey
RV survey
LAI Knowledge
Survey
Exchange
Results
Considerations for
requirements
volatility
7
Literature Background
• Most of the requirements volatility research to date has been focused
on software systems
• Various research methods have been utilized to investigate the causes
and effects of requirement volatility – a methodological breakdown of
the studies reviewed to date is below
1
Simulation Model
2
5
Survey / Model
Survey
Project data analysis
Interviews
3
2
• However, there still a lack of empirical data to determine the
quantitative impact of requirements volatility on systems engineering
effort for a broader base of engineering projects
8
Implications to COSYSMO
Reuse
Categories
Volatility
Factor
# Requirements
# Interfaces
# Scenarios
# Algorithms
Size
Drivers
• Volatility was identified as a size
driver adjustment factor in
COSYSMO [Valerdi, 2005]
• It is currently a placeholder in
the model
COSYSMO
Effort
Effort
Multipliers
- Application factors
-8 factors
- Team factors
-6 factors
Calibration
[Fortune, 2009]
9
Impact of volatility after the
requirements have been baselined
PROJECT WITH VOLATILITY
PROJECT WITH LOW VOLATILITY
Number of Requirements Over Time
1800
1800
1600
1600
1400
1400
# Req.
# Req.
Number of Requirements Over Time
1200
1000
800
600
SRR
1600
1400
1400
Delta SE
effort
SE Hours
SE Hours
1600
1200
1200
800
600
600
SRR
Delta SE
effort
1000
800
400
SRR
Systems Engineering effort
Systems Engineering effort
1000
Planned # Req
Actual # Req
1000
Planned # Req
Actual # Req
800
600
1200
400
SRR
10
Observations
1.
Requirements volatility is correlated with an increase in the size drivers of systems
engineering effort [Valerdi, 2005; Houston,2000]
2.
Requirements Volatility is expected during the early stages of a project
(conceptualize / requirements phase). It becomes a concern when it ocurrs after
the requirements phase is complete because it is likely to result in re-work of
engineering products [Kotonya and Sommerville (1995); Rosenberg and Hyatt (1996); Ferreira, (2002)]
3.
Requirements volatility is correlated to increases in cost, schedule duration and rework [Ferreira et al, 2009; Zowghi and Nurmuliani, 2002]
4.
Requirements volatility may have a compounding effect on systems engineering
effort because the changes may not only cause rework but also trigger increases in
the number of lower-level requirements [Kulk and Verhoef, 2008]
5.
The use of volatility metrics and thresholds can help determine the adequacy of SE
processes and staff and trigger corrective actions to mitigate further volatility (Roedler
and Rhodes, 2007)
6.
Requirements volatility is caused by a number of factors that are both external (e.g.
environmental, customer priority changes) and internal (process capability,
organizational policies)
Based on the review of the literature a causal model was
developed to investigate both research questions
11
Causal Model (normative)
Case 1 research question: what causes requirements volatility?
Contextual /
Environmental
changes
+
Changes in org.
structure and
process
(Jones 1994)
(Curtis et al. 1988)
Zowghi et al. (2000)
+
Customer-Driven
Changes
-
(Kotonya and
Sommerville 1998)
(Ferreira, 2002)
(GAO, 2004)
(Ferreira, 2002)
(Ferreira, et al 2009)
Requirements
volatility
(Houston, 2000)
(Ferreira 2002)
(Charette et al,2003)
(Charette et al,2003)
-
Re-work
(GAO, 2004)
+/(Houston, 2000)
Zowghi et al. (2000)
(Ferreira, et al 2009)
-
SE Process
Maturity
-
(Ferreira 2002)
Project Cost
+/-
Number of Sys
Requirements
(Ferreira 2002)
+
SE
Productivity
(Valerdi 2005)
+/-
+/-
SE Project
Effort
(Valerdi 2005)
+/-
Project
Schedule
(Ferreira, et
al 2009)
(Houston, 2000)
(Ferreira 2002)
(Charette et al,2003)
+
+
-
Requirements.
Re-use
(Valerdi 2005)
+
Quality
Technology
Maturity
+
(Jones 1994)
(Curtis et al. 1988)
(Kotonya and
Sommerville 1998)
+
Experienced
staff
+/+/-
(Valerdi 2005)
SE Cost
(Valerdi 2005)
Customer
Satisfaction
Dependent
Variable
Independent
Variable
12
Causal Model (normative)
Case 2 research question: how does requirements volatility impact
systems engineering effort?
Contextual /
Environmental
changes
+
Changes in org.
structure and
process
(Jones 1994)
(Curtis et al. 1988)
Zowghi et al. (2000)
+
Customer-Driven
Changes
-
(Kotonya and
Sommerville 1998)
(Ferreira, 2002)
(GAO, 2004)
(Ferreira, 2002)
(Ferreira, et al 2009)
Requirements
volatility
(Houston, 2000)
(Ferreira 2002)
(Charette et al,2003)
(Charette et al,2003)
-
Re-work
(GAO, 2004)
+/(Houston, 2000)
Zowghi et al. (2000)
(Ferreira, et al 2009)
-
SE Process
Maturity
-
(Ferreira 2002)
Project Cost
+/-
Number of Sys
Requirements
(Ferreira 2002)
+
SE
Productivity
(Valerdi 2005)
+/-
+/-
SE Project
Effort
(Valerdi 2005)
+/-
Project
Schedule
(Ferreira, et
al 2009)
(Houston, 2000)
(Ferreira 2002)
(Charette et al,2003)
+
+
-
Requirements.
Re-use
(Valerdi 2005)
+
Quality
Technology
Maturity
+
(Jones 1994)
(Curtis et al. 1988)
(Kotonya and
Sommerville 1998)
+
Experienced
staff
+/+/-
(Valerdi 2005)
SE Cost
(Valerdi 2005)
Customer
Satisfaction
Dependent
Variable
Independent
Variable
13
Moderating impact of expected volatility / thresholds
Org Improvement
Actions
+
SE Process
Maturity
Experienced
staff
-
+/-
+
(Ferreira, 2002)
(GAO, 2004)
-
+/-
Requirements
volatility
+/-
(Kulk and Verhoef 2008
Zowghi and Nurmuliani
(1998)
(Kulk and
Verhoef 2008)
+/-
+/-
(Valerdi 2005)
SE Project
Effort
+/-
Volatility Metrics /
Thresholds
(Houston, 2000)
Zowghi et al. (2000)
(Ferreira, et al 2009)
Number of
Requirements
+/-
(Ferreira 2002)
(Ferreira, et
al 2009)
+/-
Project
Schedule
(Valerdi 2005)
SE Cost
Dependent
Variable
Independent
Variable
Moderating
Variable
14
Questions for discussion
1. Are there other important causes of volatility missing in the model?
2. In what cases is the relationship between requirements volatility and
# of systems requirements a positive one, and in what cases is it a
negative one?
3. Should the impact of requirements volatility on the # of system
requirements be adjusted based on the criticality/coupling of the
requirements that are added/modified/deleted?
4. Does volatility have a direct impact on productivity?
5. Should volatility thresholds vary depending on the size and duration
of a project?
6. How does the lifecycle phase affect the expected level of volatility?
7. Feedback on the survey -
15
Next Steps
• Utilize recommendations/suggestions from this
workshop to update the causal model and
survey
• Administer the survey to LAI workshop
participants
• Need to develop an approach to generate cost
driver weight factors based on requirements
volatility
• Start the industry outreach to reach agreement
on data-sharing
16
References
•
Boehm, B. (1991). Software Risk Management: Principles and Practices. IEEE Software 8, 1. Pp 32-41.
•
Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in
software engineering by using analytical modeling and software process simulation.” The Journal of Systems and Software.
Vol. 82, pp 1568-1577.
•
Fortune, J. (2009). Estimating systems engineering reuse with the constructive systems engineering cost model (COSYSMO
2.0). Doctoral Dissertation. University of Southern California, Industrial and Systems Engineering Department.
•
GAO-04-393 (2004). Report to the Committee on Armed
•
Services, U.S. Senate. Defense Acquisitions. Stronger Management Practices Are Needed to Improve DOD’s SoftwareIntensive Weapon Acquisitions
•
Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of
Defense Software Engineering. Pp 20-25.
•
Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State
University
•
INCOSE Systems Engineering Handbook, Version 3, INCOSE, June 2006
•
ISO/IEC (2008). ISO/IEC 15288:2008 (E) Systems Engineering - System Life Cycle Processes.
•
Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd.
•
Kulk, G, and Verhoef, C. (2008). “Quantifying requirements volatility effects”. Science of Computer Programming. Vol 72. pp
136–175
•
MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense.
•
Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts Institute of
Technology, INCOSE, and PSM.
•
Rosenberg, L. and Hyatt L., (1996). A Software Quality Model and Metrics for Identifying Project Risks and Assessing
Software Quality. (retrieved from http://satc.gsfc.nasa.gov/support/STC_APR96/qualtiy/stc_qual.html)
•
Schunn, C. (2008) Engineering Educational Design. Educational Designer, 1(1)
•
Valerdi, R. (2005). The constructive systems engineering cost model (COSYSMO). Doctoral Dissertation. University of
Southern California, Industrial and Systems Engineering Department.
•
Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance.
17
Proceedings of the Ninth Asia-Pacific Software Engineering Conference