University of Southern California
Center for Systems and Software Engineering
Requirements Volatility
Dr. Mauricio E. Peña
February 19, 2014
1
University of Southern California
Center for Systems and Software Engineering
Agenda
• Introduction
• Definitions
• Requirements development and management
• Requirements metrics
• Causes of requirements volatility
• Impacts of requirements volatility
• Mitigation approaches
• Summary
2
University of Southern California
Center for Systems and Software Engineering
Importance of Understanding
Requirements Volatility
• Requirements volatility has been identified by numerous
research studies as a risk factor and cost-driver of systems
engineering projects1
• Requirements changes are costly, particularly in the later
stages of the lifecycle process because the change may require
rework of the design, verification and deployment plans2
• The Government Accountability Office (GAO) concluded in a
2004 report on the DoD’s acquisition of software-intensive
weapons systems that missing, vague, or changing
requirements are a major cause of project failure3
System developers often lack effective methods and tools
to account for and manage requirements volatility
Source: 1- Boehm (1991), 2- Kotonya and Sommerville (1995), 3- GAO-04-393
3
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility is Expected
• Changes to requirements are a part of our
increasingly complex systems & dynamic business
environment
– Stakeholders needs evolve rapidly
– The customer may not be able to fully specify the system
requirements up front
– New requirements may emerge as knowledge of the system
evolves
– Requirements often change during the early phases of the
project as a result of trades and negotiations
• Requirements volatility must be anticipated and
managed
Sources; Kotonya and Sommerville (1995); Reifer (2000)
4
University of Southern California
Center for Systems and Software Engineering
Requirements Definitions
“Requirements are intended to change vague desires into explicit
and unambiguous statements of what the customers want” 1
“A requirement is a capability that a system must supply or a
quality that a system must possess in order to solve a problem or
achieve an objective within the system’s conceptual domain” 2
“A requirement is: (1) A condition or capability needed by a user
to solve a problem or achieve an objective (2) A condition or
capability that must be met or possessed by a system or system
component to satisfy a contract, standard, specification, or other
formally imposed documents ” 3
Source: 1- Weinberg, G. (1983), 2- Costello and Liu (1995); 3- IEEE (1990)
5
University of Southern California
Center for Systems and Software Engineering
When is that Systems
Engineer going to tell me
the S/W requirements?
6
University of Southern California
Center for Systems and Software Engineering
Requirements Categories
1. Customer or operational requirements define the expectations
of the system in terms of operational scenarios and
environments, mission objectives, and measures of
effectiveness
2. Functional requirements specify what has to be done in terms
of tasks or activities
3. Performance requirements define how well or to what extent
the activities must be accomplished
4. Design requirements dictate how to build products and what
manufacturing processes to follow
5. Derived requirements are spawned from higher-level
requirements; and
6. Allocated requirements: high level-requirements are
partitioned into multiple lower-level requirements
Source: DoD Systems Management College (2001)
7
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility Definitions
• Requirements volatility is typically defined as the
change in requirements (added, deleted, and
modified) over a given time interval
• Also known as:
• Requirements creep: An increase in scope and
number of system requirements
• Requirements churn: Instability in the requirements
set – requirements are modified or re-worked without
necessarily resulting in an increase in the total
number of requirements
Source: MIL-STD-498
8
University of Southern California
Center for Systems and Software Engineering
Requirements Management
• The Systems Engineering Capability Model (EIA, 2002) defines
five activities that influence the generation and evolution of
requirements1
– Problem refinement
– Requirements analysis
– Requirements quality
– Requirements evolution and maintenance
– Feedback and verification
• Requirements definition and analysis are sometimes
collectively referred to as requirements engineering:
– The use of systematic and repeatable techniques to identify,
document, develop and maintain a complete and consistent
requirement set2
Source: 1- EIA (2002); 2- Kotonya and Sommerville (1998)
9
University of Southern California
Center for Systems and Software Engineering
Context Diagram – Requirements
Analysis Process
Source: Systems Engineering Handbook. INCOSE Version 3.2
10
University of Southern California
Center for Systems and Software Engineering
Requirements Development
The customer’s needs
and objectives are
developed into
operational concepts to
satisfy them
System functional and
performance requirements
are developed to define
what the system must do
and under what
environments and
constraints it must
perform
Once the baseline system
requirements and system
architecture are
established, they are
decomposed through
successively lower levels.
Source: DoD Instruction 5000.02 (2008)
11
University of Southern California
Center for Systems and Software Engineering
Requirements Baseline and
Change Control
• A Requirements baseline is established when the functional
requirements and characteristics of the system are formally
documented and placed under configuration control
– After the System Requirements Review or equivalent milestone
– Modifications to the baseline are requested through Engineering
Change Proposals (ECP)
• Typically, a formal management approval structure is
established to prioritize the ECPs, evaluate their impact, and
make decisions regarding their approval and implementation
– Other change documents used to control the requirements
baseline are Requests for Deviation and Requests for Waivers
– RDWs propose a departure from the baseline by allowing the
acceptance of products that are non-conformant or do not meet
the requirements as stated
Source: DoD Instruction 5000.02 (2008)
12
University of Southern California
Center for Systems and Software Engineering
Systems Engineering Reviews and
Acquisition Lifecycle Phases
User Needs
Evolutionary Acquisition or Single Step to
Full Capability
Acquisition Milestones
A
Materiel
Solution
Analysis
(Program
Initiation)
B
Technology
Development
Engineering and
Manufacturing
Development
Pre-Systems Acquisition
Tech Reviews
ASR
IOC
C
Production &
Deployment
Systems Acquisition
SRR SFR PDR CDR
FOC
Operations &
Support
Sustainment
FCA PCA
Requirements Review
Documents
Sys Perf. Spec.
Draft
Item Perf. Specs.
Item Detail
Detailed Design
Rqmts.
CI
System
Definition
Item Design
Rqmts.
SYS
Sys Tech
Requirements
Source: DoD Instruction 5000.02 (2008)
13
University of Southern California
Center for Systems and Software Engineering
Requirements and Evolutionary
Acquisition Process
Source: DoD Instruction 5000.02 (2008)
14
University of Southern California
Center for Systems and Software Engineering
Requirements Base Measures
Base Measures
Measurement Method
# of Requirements
Count the # of requirements in the
applicable specification (# of
shalls/wills/musts)
# of requirements defects
Count the # of defects
# of requirements changes
Count the # of changes per category
(added, deleted or modified) from
Engineering Change Proposals (ECP),
requirements database, etc.
Impact of each change
Estimate effort-hours per change
(sometimes estimated in ECPs,
budgeted hours vs. actual hours)
Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010
15
University of Southern California
Center for Systems and Software Engineering
Requirements Derived Measures
% Requirements Growth =
((# of requirements in current baseline - # of requirements in previous baseline)
( # of requirements in previous baseline) X 100
% Requirements Modified =
(# of requirements modified / total # of requirements) X 100
Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010
16
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility Metrics (1 of 2)
Program “A” Requirements Volatility
% of total Requirements Changing
100%
90%
Total Changes
Modified
Projected Trend
Added
Deleted
80%
70%
60%
Planned
SRR
MetricDriven
SRR
50%
40%
30%
20%
10%
0%
Time
Volatility expressed as a % of the total number of requirements
Systems Engineering Leading Indicators Guide, Version 2.0, 2010
17
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility Metrics (2 of 2)
Number of Requirements
Program “B” Requirements Volatility
50
Added
Modified
Deleted
40
30
20
10
0
Time (months)
Volatility metrics track the number and type of changes over time
Source: Hammer, T., Huffman, L., and Rosenberg, L. (1998).
18
University of Southern California
Center for Systems and Software Engineering
Periodic and Cumulative Measures
# of Requirements Changes
180
160
Req. Added
140
Req. Deleted
120
Req. Modified
100
Cum. Deleted
Cum. Modified
80
Cum. Added
60
Cum. Changes
40
20
0
Conceptualize
F M
J
Development
A M J
Operational Test &
A S
JEvaluation
D
N to
OTransition
Operation
19
University of Southern California
Center for Systems and Software Engineering
Observations on the Behavior of
Volatility
1. Requirements volatility is caused by an identifiable
set of project and organizational factors
2. The level of requirements volatility is a function of
the system life cycle phase
3. Requirements volatility leads to an increase in
project size and cost
4. The cost and effort impact of a requirements
change increases the later the change occurs in the
system life cycle
5. The impact of requirements volatility varies
depending on the type of change: added, deleted, or
modified
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston
(2000); Zowghi and Nurmuliani (2002); Ferreira (2002); Kulk and Verhoef 2008
20
University of Southern California
Center for Systems and Software Engineering
Expected Level of Volatility Across
Lifecycle Phases (Survey)
don't know
20
<5%
5-10%
don't know
don't know
don't know
<5%
5-10%
<5%
Number of Respondents
10-20%
15
<5%
10-20%
10
5-10%
>20%
5
5-10%
>20%
10-20%
0
>20%
Conceptualize
Operational Test &
Evaluation
Expected Requirements Volatility (%)
Development
10-20%
>20%
Transition to
Operation
21
University of Southern California
Center for Systems and Software Engineering
Impact of Hardware/Software Project
Breakdown on Expected Volatility
Operational Test & Evaluation Lifecycle Phase
Transition to Operation Lifecycle Phase
100%
% of Respondents per Category
% of Respondents per Category
100%
80%
60%
40%
20%
0%
Respondents from S/W
intensive projects tend
to expect more volatility
later in the lifecycle
80%
60%
40%
20%
0%
<5%
5-10%
10-20%
>20%
<5%
Expected Requirements Volatility (%)
75% Hardware, 25% Software
25% Hardware, 75% Software
50% Hardware, 50% Software
5-10%
10-20%
>20%
Expected Requirements Volatility (%)
75% Hardware, 25% Software
25% Hardware, 75% Software
Source: LAI Knowledge Exchange / CSSE ARR Survey (2010)
50% Hardware, 50% Software
22
University of Southern California
Center for Systems and Software Engineering
Sample Volatility Profiles
% of Requirements, Added, Deleted or Modified
50%
45%
#1
#2
#3
#4
40%
Participant inputs
#5
#6
#7
Localized peaks in volatility
35%
30%
25%
20%
15%
10%
5%
0%
Conceptualize
Development
Operational Test &
Evaluation
Source: Practical Software & Systems Measurement Conference Survey (2010)
Transition to
Operation
23
University of Southern California
Center for Systems and Software Engineering
Requirements Trends as a Leading
Indicator of Project Performance
• It helps to determine the
stability and completeness of
the system requirements
which could potentially
impact project performance
Requirements Growth Trends
Corrective
Action Taken
Number of Requirements
• Evaluates trends in the
growth, change,
completeness and
correctness of the system
requirements.
Planned # of
Requirements
Actual # of
Requirements
Projected # of
Requirements
SRR
PDR
Source: Systems Engineering Leading Indicators Guide, Version 2.0, 2010
Time
CDR
24
University of Southern California
Center for Systems and Software Engineering
Causes of Requirements Volatility
• External factors
–
–
–
–
–
Shifting customer priorities and needs
Changes in the political and business environment
Addition or change of stakeholders
Development of new technologies
Changes in co-dependent systems
• Internal factors
– Deficient requirements development processes
– Lack of experienced systems engineering resources applied to
requirements analysis
– Poor initial understanding or interpretation of customer needs by
the development team
– Changes in organizational structure and policies
Sources: Kotonya and Sommerville (1995); Houston (2000); Zowghi and Nurmuliani (2002); Kulk and
Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)
25
University of Southern California
Center for Systems and Software Engineering
Systems Dynamics View of the Causes of
Requirements Volatility
6
7
Contextual /
Environmental
Changes
Experienced
Staff
2
1
Poor
Understanding of
the System &
Customer Needs
8
3
Customer-driven
Scope Change
Changes in
Co-dependent
Systems
9
Changes in
COTS Products
4
Requirements
Process
Maturity
Changes in Org
/ structure/
Process
5
Technology
Maturity
Requirements
Volatility
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston
(2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)
26
University of Southern California
Center for Systems and Software Engineering
Impacts of Requirements Volatility
•
Several research studies have found that requirements
volatility is positively correlated with increase in
•
The functional size of the project (due to discarded lines of code
and additional requirements)
•
Engineering effort
•
Cost (often >20%) and schedule duration
•
Changes in requirements result in additional rework and
increased defect density
•
The impact of changing requirements is much greater the later
the change occurs in the project lifecycle
•
Removing a requirement may not necessarily result in a net
decrease in engineering effort and cost
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston
(2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)
27
University of Southern California
Center for Systems and Software Engineering
Systems Dynamics View of the Impact of
Requirements Volatility
Quality
Rework /
Defects
Project Cost
Requirements
Volatility
+/-
+/-
Customer
Satisfaction
Project Effort
+/-
Number of
System
Requirements
+/+/-
Project
Schedule
+/-
Sources: Kotonya and Sommerville (1995); Hammer et al. (1998); Malaiya and Denton (1999); Stark et al. (1999); Houston
(2000); Zowghi and Nurmuliani (2002); Kulk and Verhoef 2008; Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009)
28
University of Southern California
Center for Systems and Software Engineering
Cost Commitment on Projects
C om m itm ent to Technology,
Configuration, P erform ance, Cost, etc.
%
100
Cost Incurred
75
S ystem -S pecific Know ledge
50
25
E ase of C hange
N
E
E
D
ConceptualPrelim inary
Design
Detail Design
and
Developm ent
Construction
and/or
P roduction
S ystem Use, P haseout,
and D isposal
Changes to the System are more difficult to implement the
later they occur in the lifecycle
Source: Blanchard and Fabrycky (1998)
29
University of Southern California
Center for Systems and Software Engineering
Systems Engineering Effort Penalty Due to Volatility
Expected Cost Penalty due to
Requirements Volatility (Survey Results)
11.0
10.0
9.0
8.0
7.0
6.0
5.0
4.0
3.0
2.0
1.0
0.0
Conceptualize
Development
Operational Test
& Evaluation
Transition to
Operation
Source: COCOMO Forum Survey (2010) and PSM Workshop (2011)
30
University of Southern California
Center for Systems and Software Engineering
Expected Cost Penalty per Change
Category (Survey Results)
Systems Engineering Effort Penalty
due to Volatility
10.5
added
deleted
9.0
modified
7.5
6.0
4.5
3.0
1.5
0.0
-1.5
Conceptualize
Development
Operational Test &
Evaluation
Transition to
Operation
Source: COCOMO Forum Survey Exercise (2010) and PSM Workshop (2011)
31
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility: A Sizing and
Cost Estimation Challenge
• Our dynamic competitive environment leads to rapid
changes in objectives, constraints and priorities
• Requirements are often emergent instead of prespecifiable
• These trends have lead to the increasing use of
incremental and evolutionary development strategies
• Counting the number of source lines of code (SLOC)
delivered may underestimate the size of the system
– Portions of the code may have been modified or deleted due
to changing requirements
– These modified or discarded lines would not result in an
increase in the final SLOC count
Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft)
32
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility and Parametric
Cost Estimation
• The Constructive Cost Model (COCOMO II) incorporated the
impact of requirements volatility on the effective size of
software systems using an adjustment factor called REVL
(Requirements Evolution and Volatility)
• REVL is also known as the “breakage parameter,” defined as
the % of code discarded due to changes in requirements.
 REVL 
Size  1 
  SizeD
100 

Where,
SizeD is the initial equivalent size of the software product adjusted
for reuse
Source: Boehm et al. (2000).
33
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility Expected Ranges
Rating Level
RVOL Range
RVOL Average
Very Low
0-6%
3%
Low
6-12%
9%
Nominal
12-24%
18%
High
24-48%
36%
Very High
>48%
72%
Source: AFCAA Software Cost Estimation Metrics Manual Version 0.82 (draft)
34
.04   Wi
MM
 2.8KDSI 1University
of Southern California
MM nom
i 1
nom  2.8KDSI 
Center for Systems and Software Engineering
1.20
4
Requirements Volatility in Ada COCOMO
(1 of 2)
• Ada process model attempted to reduce the
diseconomies of scale (exp = 1.20) of the COCOMO
embedded mode
MMnom  2.8KDSI
1.20
Where,
MMnon = number of man-months required to develop a
nominal software product
KDSI = thousands of delivered source instructions
Source: Ada COCOMO and the Ada Process Model (Boehm and Royce, 1989)
35
University of Southern California
Center for Systems and Software Engineering
Requirements Volatility in Ada COCOMO
(2 of 2)
•
Improvements in diseconomies of scale resulting
from the Ada process model by
1.
2.
3.
4.
Early software architecture definition
Mitigation of risks by PDR
Requirements stabilization by PDR; and
The team’s experience with the Ada process model
MM nom  2.8KDSI 
1.04 
•
•
4
Wi
i 1
Each of these four elements is rated using a scale
from 0.00 to 0.05
The ratings are summed to determine Σ
Source: Ada COCOMO and the Ada Process Model (Boehm and Royce, 1989)
36
University of Southern California
Center for Systems and Software Engineering
Ada COCOMO Σ factor:
Requirements Volatility
No Changes
Small Noncritical
Changes
Frequent
non-critical
Changes
Occasional
Moderate
Changes
Frequent
Moderate
Changes
Many Large
Changes
.00
.01
.02
.03
.04
.05
System requirements baselined, under
rigorous change control
Fully
Mostly
Generally
Some
Little
None
Level of uncertainty in key requirements
areas, mission, user interfaces,
hardware, other interfaces
Very Little
Little
Some
Considerable
Significant
Extreme
Organizational track record in keeping
requirements stable
Excellent
Strong
Good
Moderate
Weak
Very Weak
Use of incremental development to
stabilize requirements
Full
Strong
Good
Some
Little
None
System architecture modularized
around major sources of change
Fully
Mostly
Generally
Some
Little
None
Characteristic
Rating Scale
Source: Ada COCOMO and the Ada Process Model (Boehm and Royce, 1989)
37
University of Southern California
Center for Systems and Software Engineering
Actions to Mitigate Requirements
Volatility
1. Improve communication between stakeholders and
development team
2. Improve the capability and maturity of the
requirements development process
3. Additional training / more experienced personnel
4. Increase the use of prototypes and models to
validate requirements
5. Improve technology insertion processes
6. Increase management oversight (i.e. metrics,
reviews)
7. Establish a more strict change control process
38
University of Southern California
Center for Systems and Software Engineering
Mitigation Factor Effectiveness:
Survey Results
Source: CSSE ARR Survey (2010)
39
University of Southern California
Center for Systems and Software Engineering
Characteristics of Good
Requirements (1 of 3)
1. Necessary
•
Avoid specification of design details & redundant requirements
2. Implementation Independent
•
Specify “what” is to be done, not “how” to do it
3. Clear and Concise
•
Don’t use ambiguous terms; avoid compound requirements with
multiple “shall statements” and conjunctions such as “and”, “or”
•
Example – is the following a good requirement?
•
“The car shall accelerate from 0 to 60 MPH in 5 seconds and
break to a standstill in 10 seconds under extreme conditions”
Source: Systems Engineering Handbook. INCOSE Version 3.2
40
University of Southern California
Center for Systems and Software Engineering
Characteristics of Good
Requirements (2 of 3)
4. Complete
•
Provide all the information necessary for the developer to
understand what you mean
•
Do you have any TBXs (to be determined, to be confirmed)?
5. Consistent
•
Is the requirement self-consistent?
•
Do you have conflicting statements?
6. Achievable
•
Engage the developer/designer in the requirements definition to
ensure they concur that the requirement can be met
Source: Systems Engineering Handbook. INCOSE Version 3.2
41
University of Southern California
Center for Systems and Software Engineering
Characteristics of Good
Requirements (3 of 3)
7. Traceable
•
Are there any orphan requirements (i.e. they don’t trace to a
higher level requirement)?
8. Verifiable
– Avoid subjective terms such as “provide support,” and “user
friendly”
– Look out for conditions that are not clearly achieved such as
“minimize,” “to the degree possible,” etc.
•
Example:
1. The aircraft should be as light as possible
2. The wet weight of the aircraft shall not exceed 2000 lbs.
Source: Systems Engineering Handbook. INCOSE Version 3.2
42
University of Southern California
Center for Systems and Software Engineering
Summary
• Requirements changes are unavoidable in today’s
rapidly changing environment
• Some level of volatility is healthy and is part of the
system development process
• However, excessive volatility and late requirements
changes can have a very detrimental effect on the
project – this is a risk that must be addressed and
mitigated
• Requirements volatility should be anticipated,
managed, and accounted for in cost and size
estimates
43
University of Southern California
Center for Systems and Software Engineering
References
• Air Foce Cost Analysis Agency Software Cost Estimation Metrics Manual, Draft Version 0.82 (2011)
• Boehm, B. and Royce, W. (1989). Ada COCOMO and the Ada Process Model, TRW Defense Systems Group
• Boehm, B. (1991). “Software Risk Management: Principles and Practices.” IEEE Software. Vol 8 (No.1).pp 32-41
• Boehm, B., Abts, C., Brown, A.W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D.J., and Steece, B. (2000). Software Cost Estimation with
COCOMO II. Prentice Hall
• EIA (2002). EIA-731.1 - Systems Engineering Capability Model
• Ferreira, S., Collofello, J., Shunk, D., and Mackulak, G. (2009). “Understanding the effects of requirements volatility in software engineering by using
analytical modeling and software process simulation.” The Journal of Systems and Software. Vol. 82, pp 1568-1577.
• General Accounting Office (2004). Stronger Management Practices are Needed to Improve DOD’s Software-intensive Weapon Acquisitions (GAO-04-393).
Defense Acquisitions.
• Hammer, T., Huffman, L., and Rosenberg, L. (1998). “Doing requirements right the first time.” Crosstalk, the Journal of Defense Software Engineering. Pp
20-25.
• Houston, Dan X. (2000). A Software Project Simulation Model for Risk Management, Ph.D. Dissertation, Arizona State University
• IEEE (1990). IEEE Standard Glossary of Software Engineering Terminology (ANSI), [1-55937-067-X] [SH13748-NYF], IEEE
• INCOSE (2010). Systems Engineering Handbook. INCOSE Version 3.2
• Kotonya, G., Sommerville, I., (1998). Requirements Engineering: Processes and Techniques. John Wiley and Sons, Ltd.
• MIL-STD-498. 1994. Software Development and Documentation. U.S. Department of Defense.
• Nguyen, V. and Boehm, B. (2010). A COCOMO Extension for Software Maintenance. 25th International Forum on COCOMO and Systems/Software Cost
Modeling
• Reifer, Donald J. 2000. “Requirements Management: The Search for Nirvana.” IEEE Software. Vol 17 (No. 3), pp 45-47
• Roedler, G. and Rhodes, D. (2007). Systems engineering leading indicators guide. Version 1. Massachusetts Institute of Technology, INCOSE, and PSM
• Weinberg, G. (1993). Quality Software Management: Volume 2 First-Order Measurement. Dorset House.
• Zowghi, D. and Nurmuliani, N. (2002). A Study of the Impact of Requirements Volatility on Software Project Performance. Proceedings of the Ninth AsiaPacific Software Engineering Conference
44