Requirements Volatility
Workshop
Rick Selby
USC Center for Systems & Software Engineering
http://csse.usc.edu
February 15, 2007
Attendee List
Rick Selby
JoAnn Lane
Sue Koolmanojwong
Scott Rigby
Mike Barker
Jim Lambert
Tom Schroeder
Robert Culbertson
Gary DeGregoro
Ray Hunnicutt
NGC
USC CSSE
USC CSSE
Raytheon
NAIST
CISCO
BAE Systems
CISCO
Motorola Labs
Lockheed Martin
Rick.selby@ngc.com
Jolane@usc.edu
Koolmano@usc.edu
Scott_rigby@raytheon.com
Mbarker@computer.org
Jlamber@cisco.com
Tom.schroeder@baesystems.com
Rculbert@cisco.com
Gary.degregorio@motorola.com
Ray.hunnicutt@lmco.com
Karen Owens
Da Yang
The Aerospace Corp.
Institute of Software CAS
Karen.Owens@aero.org
Yangda@itechs.iscas.ac.cn
Attendee List
Dan Ligett
Ray Madachy
Anna Warner
Barbara Park
A Winsor Brown
George Hulin
Rosaline Lewis
Barry Boehm
Di Wu
Vu Nguyen
Rod Robertson
Softstar
USC CSSE
Boeing
Boein
USC CSSE
DSC
The Aerospace Corp.
USC CSSE
USC CSSE
USC CSSE
Boeing
Ligett@softstarStystems.com
Madachy@usc.edu
Anna.warner@boeing.com
Barbara.park@boeing.com
Awbrown@usc.edu
g.huling@alumni.duke.edu
Rosalind.lewis@aero.org
Boehm@usc.edu
Diwu@usc.edu
Nguyenvu@usc.edu
Rodney.robertson@boeing.com
Workshop Guidelines
 Product: briefing, preferably with notes
 Topics should include:
– Most critical success factors in each area
– Current best practices for addressing them
– Areas for further research
 Rated 0-10 on value and difficulty of
research
4
Requirements Critical Success Factors
 Technical
– (tech)Investigate req interdependencies and dependencies
– (tech)Demistify the notion that requirements are independent
– (tech)Requirements gathering-what are the important attributes
that should be captured
– (tech)Want virtual model and understand attributes
– (tech and people)Better ways to prioritize requirements,
considering risk, cost factors, stakeholder utility function and how
to reconcile stakeholders interest
– (tech and people)Same as above with greater emphasis on
change
– (tech)Specify requirements/not implementation; users often want
to specify implementation
– (tech)What are the critical/driving requirements
5
Requirements Critical Success Factors (continued)
 Technical
– (tech)Relate requirements work back to COSOSIMO and
COSYSMO
– (tech)Missing rich enough vocabulary to model “this whole thing”
(wants, needs, problems, issues, business drivers that are
referred to as requirements)
– (tech)Need modeling approach other than English
– (tech/tool)How do we automatically measure requirements,
requirement volatility, progress, scope
– (tech)Accurately assess/predict impact of changes
– (tech)Standard classification method to capture types of
requirements and level of requirements
– (tech)Classify requirements volatility types and define metrics to
measure volatility
– (tech)Address differences in req vol across market segments
6
Requirements Critical Success Factors (continued)
 Technical
– (tech)Function points necessary but not sufficient—
miss technical performance and constraints
– (tech)Capabilities versus requirements
– (tools/tech)Tools: relate estimation inputs to the
outputs (for example, inputs req lead to sloc output
instead of sloc to effort)
– (tech)Global collaboration—state requirements in a
way that they can be understood globally and
implemented locally
– (tech)Specify the behavior for off-nominal conditions
and events
– (tech)Detect the confliction between requirements
7
Requirements Critical Success Factors (continued)
 Tool
– (tech/tool)How do we automatically measure
requirements, requirement volatility, progress, scope
– (process/tool)Requirements traceability from the
beginning (who initiated and why, which application,
which test)
– (tools)Investigate tools available for requirements
process (measurement, tracking)
– (tools/tech)Tools: relate estimation inputs to the
outputs (for example, inputs req lead to sloc output
instead of sloc to effort)
– (process/tools)Need a good way to move back and
forth between reqs and models
8
Requirements Critical Success Factors (continued)
 Product
– (prod)Define requirements in a sufficient enough way
to cost them, but high level enough to provide
flexibility (R. Turner comment)
9
Requirements Critical Success Factors (continued)
 Process
– (process)Capture requirement up front using elicitation in a way
that avoids future change
– (process)Plan for requirements to change
(evolutionary/incremental development)
– (process)Late binding of requirements-need to specify in a way
to promote late binding
– (process)Involve all key stakeholders and ensure their
accountability for the requirements
– (people/process)Treat requirements process as a negotiation
and learning process
– (management/process)Minimize requirements volatility by
managing decision volatility
– (management/process)Link decisions and requirements
– (process)Need to identify assumptions as well as requirements
10
Requirements Critical Success Factors (continued)
 Process
– (process)Be clear what is an assumption and what is a
requirement
– (people/process)Treat requirements process as a negotiation
and learning process
– (process)Standard on requirements volatility-how to measure,
what to measure
– (process/methodology)Look at process used to development
requirements, especially across enterprises
 Modeling and sim
 Environment (process, tools, people)
 Instrumentation
 Increase propensity to accept changes in the requirements
process
– (process)Steal from agile, tests are one of the better forms for
capturing requirements
11
Requirements Critical Success Factors (continued)
 Process
– (process/management)Get immediate feedback from
development and test teams that requirements can be
implemented and tested
– (process/management)Write requirements in
alignment/context with business drivers
– (process/tool)Requirements traceability from the
beginning (who initiated and why, which application,
which test)
– (process/management)Keeping the multiple
stakeholders engaged across the life cycle
– (process/management)Integrate measurement and
modeling of requirements changes
12
Requirements Critical Success Factors (continued)
 Process
– (management/process)Improve/investigate
methodology for requirements development by
incorporation of meta models, semantics, and
ontology
– (process)Using value based systems engineering we
need to identify the 10-20 key driving requirements for
large scale systems
– (people/process)Collaboration with customer to
prevent later volatility (using analog of collaborative
design in the requirements space)
13
Requirements Critical Success Factors (continued)
 Process
– (process)Moving from requirements to models too
early leads to premature codification of solution(s)
– (process/tools)Need a good way to move back and
forth between reqs and models
– (process)25% req change results in 100% change in
the solution space
– (process)Requirements completeness for context at
various points in time
14
Requirements Critical Success Factors (continued)
 People
– (tech and people)Same as above with greater
emphasis on change
– (people) Committed, representative, available,
collaborative, knowledeable (CRACK) stakeholders
– (people/process)Treat requirements process as a
negotiation and learning process
– (people/process)Treat requirements process as a
negotiation and learning process
– (people)Make stakeholder incentives explicit
– (people)Function point modeling to provide another
way to look at requirements
15
Requirements Critical Success Factors (continued)
 People
– (people)Data modeling (part of FP modeling) to
analyze requirements
– (people/management)Change management and
expectation management
– (people/org)How do we convince customers to think in
smaller steps
– (people/org)Customers tend to think 5-10 yrs in the
future, others tend to think 9 months ahead
– (people/process)Collaboration with customer to
prevent later volatility (using analog of collaborative
design in the requirements space)
– (people)Experience of team
16
 People
– (people)Diverse representation in the systems engineering
organization: role, function, people, cultures
– (people/org)Managing stakeholder continuity/volatility/high
variance of req interpretation
– (people)Ensure that you understand your customer (familiar with
customer)
– (people)Impact of cognitive makeup of req candidates using
standard psych tests (Toyota evaluates critical thinking, problem
solving, decision making—40 hours of testing prior to
interviewing)
– (people)UML never represent customer space, they could, but
they don’t—the first one you see is taken as a design model (any
time you make an abstract model, it is assumed by others that it
is how you plan to design the solution)—how do you convince
people that there are abstract models that are useful to define the
problem without defining the solution and to treat them as such
17
Requirements Critical Success Factors (continued)
 People
– (people)People involved in eliciting and analyzing
requirements
 Well versed in domain and the trade space for
solutions leads to more effectiveness in eliciting
and analyzing requirements
– (people)Having domain knowledge and experience in
extracting requirements leads to
18
Requirements Critical Success Factors (continued)
 Management
– (management/process)Minimize requirements volatility
by managing decision volatility
– (management/process)Link decisions and
requirements
– (management)Use models to understand effect of
requirements change on the process
– (people/management)Change management and
expectation management
– (process/management)Get immediate feedback from
development and test teams that requirements can be
implemented and tested
19
Requirements Critical Success Factors (continued)
 Management
– (process/management)Write requirements in
alignment/context with business drivers
– (process/management)Keeping the multiple
stakeholders engaged across the life cycle
– (process/management)Integrate measurement and
modeling of requirements changes
– (management/process)Improve/investigate
methodology for requirements development by
incorporation of meta models, semantics, and
ontology
20
Requirements Critical Success Factors (continued)
 Methodology
– (process/methodology)Look at process used to
development requirements, especially across
enterprises
 Modeling and sim
 Environment (process, tools, people)
 Instrumentation
 Increase propensity to accept changes in the
requirements process
21
Requirements Critical Success Factors (continued)
 Organizational
– (people/org)How do we convince customers to think in
smaller steps
– (people/org)Customers tend to think 5-10 yrs in the
future, others tend to think 9 months ahead
– (people/org)Managing stakeholder
continuity/volatility/high variance of req interpretation
22
Requirements Critical Success Factors (continued)
 (citation)Software requirements: the Requirements Set
(Ferdinandi) (about categories of requirements)
23
Research Projects
Vc
 V: 1,7,2 D:2,7,2 Establish relationships between capabilities and
requirements 3
 V: 5,6,2 D: 0,4,5 How to support client/customer in requirements
elicitation 4
– for naïve/non-expert clients
– when the elicitor does not have domain knowledge
 V: 2,5,5 D: 0,10,0 Define and evaluation a taxonomy of knowledge
needed in req elicitation 2
 A) V: 11,3,0 D: 0,8,6 Develop improved/standardized methods for
measuring and evaluating reqs/reqs changes and evaluate the
benefits on experimental projects 7, 22/14=1.57
 V: 7,7,0 D: 1,8,4 Investigate use of boundary objects for modeling
requirements in collaborative teams 5
 V: 5,6,4 D: 0,5,9 Build tools to visualize and investigate the
relationships between decisions and requirements, with emphasis on
managing changes that are resulting from decision change 5
24
Research Projects (continued)
 B)Define set of rules/metrics that can accurately
predict/reflect 13 ; D: 0,4,13 21/17=1.24
– requirements quality
– decision quality
– Req traceability with code and design (bi-directional)
– Requirements completion
– Requirements omissions
 Develop checklists/tools help one identify missing reqs 4
 Translate English requirements set into a virtual
representation 4
 C)Develop knowledge management tool to categorize
requirements and based on this, develop cost
estimate/risk predictions 6; D: 0,4,11 19/15=1.27
 Make function point models into a useful form of
boundary objects 1
25
Research Projects (continued)
 D) Define a representation and accompanying new tools
for reqs that enables automated measurement and
analysis 6; D: 0,3,11 17/14=1.21
 Investigate team org and roles that are effective at reqs
elicitation, analysis, pruning, and prioritization 5
 E) Extend USC COCOMO research to relate reqs as
inputs to SLOC as output and verify on real projects 6;
D:0,7,8 22/15=1.47
 Investigate levels of requirements and develop factors to
estimate number of reqs at various levels (requirements
normalization to get consistent count of reqs at a given
level) 4
 Investigate the effects of communicating reqs across
cultures, especially globally 3
26
Research Projects (continued)
 Small to medium projects: does a very structured process have an
impact on 5
– Req quality and req volatility
– quality and cost of end product
 Define and develop helper for ensuring that behavior for undesirable
events and off-nominal conditions are captured in the reqs 1
 Tool to identify potential emergent behaviors resulting from
requirements 3
 F) Develop an approach for costing by capabilities instead of
requirements 7; D:0,6,8 20/14=1.42
– Want to build a system “like that one” – reqs def by analogy
 Investigate feasibility of function points as an alternative to
COSYSMO reqs input 2
 Define/develop Tools to identify “similar requirements” to enable
detect redundancy, conflicts, and opportunities for reuse 5
27
Research Projects (continued)
 Compare contribution of people vs. process to the quality
of the developed requirements 3
 Define and develop a way to measure the quality of
developed requirements 3
 G) Develop meta model, semantics (categorization of
terms, naming conventions, nomenclature) and
ontologies of requirements 7; D:0,8,6 22/14=1.57
 Investigate the efficiencies of various requirements
derivation techniques for deriving high quality/complete
requirements 3
 H) Define the minimum required steps for lean
requirements-related processes 7; D:4,9,2 32/15=2.13
– Small projects
– Large projects
28
Research Projects (continued)
 Develop tool to capture requirements associated with
selected/reused/COTS components 1
 Identify preferred methodology for incorporation of legacy
requirements into new projects 2
 Investigate the extent to which we can simulate the user’s problem
space (virtual model)—executable model 3
 I) Define a mechanism for capturing reqs using test procedures (testcentric requirements development) 6; D: 0,12,1 25/13=1.92
 Define more specific completeness criteria for passing LCO/LCA
milestones 4
– Investigate continuous milestones with continuous/tightened
criteria
 Identify and do good write-ups of case studies on outstanding
requirements and processes in industry. 5
 Identify best practices for reqs development 4
 Identify preferred approach for seamless incorporation of se and sw
eng life cycles, with emphasis on software life cycle within se life
cycles 3
29
Research Projects (continued)
 Define and develop an incremental sys eng life cycle
model 3
 J) Develop a model to predict reqs volatility 14; D: 0,3,13
19/16=1.19
– Based on a given set of reqs, problem domain, and
stakeholders
– With emphasis on the velocity with which reqs were
developed, changes made, defects injected and
defected
 Capture the types and provide predictors for latent req
defects 1
30
Top 10 Research Project
 Develop a model to predict reqs volatility 14
– Based on a given set of reqs, problem domain, and stakeholders
– With emphasis on the velocity with which reqs were developed,
changes made, defects injected and defected
 Define set of rules/metrics that can accurately predict/reflect 13
– requirements quality
– decision quality
– Req traceability with code and design (bi-directional)
– Requirements completion
– Requirements omissions
 V: 11,3,0 D: 0,8,6 Develop improved/standardized methods for
measuring and evaluating reqs/reqs changes and evaluate the
benefits on experimental projects 7
 Develop knowledge management tool to categorize requirements
and based on this, develop cost estimate/risk predictions 6
31
Best Practices:
 What are the current best practices for addressing risk
related to requirements volatility
– Postponement of requirements
– Iterative requirements development
– Clarity of requirements
– Prioritize requirements
– Prioritize by feasibility
– More agile process
– Capture justification and history of requirements
– Architect product to withstand change
– Better standards for requirements documentation
– Agile prototyping
– Involve many stakeholders, especially downstream
customers
– Hierarchical classifications (Layered—577 MBASE)
32
Best Practices (continued)











Prioritize early and at high levels on the requirements hierarchy
Change propagation and traceability
Requirements traceability
Over-engineer (just in case) and create a resilient architecture that
anticipates change
Visible and understandable reqs
Measure requirements and volatility
Assess and communicate impact to all stakeholders
Training
– Processes
– Domain
Checklists
Make explicit win conditions and incentives of stakeholders
Triage requirements—if some reqs change, it does not matter, if
others change, it matters a lot
33
Best Practices





ROI analysis for alternative requirements
Use tools
Rely on standards to a larger extent
Conduct sensitivity analysis up front
Categorize requirements, capabilities, project,
performance, interface, evolution, future
34
Ease / Value
Project Letter
Research Project desc.
Difficulty
main Key Words, Phrases and Concept
(starts with)
3
2
Easy Medium
EASE
Value:
1
TopTen
HardWeighted Normalized Votes
aDevelop improved ...
Measure & evaluate; reqts/changes; assess benefits
0
8
6
22
1.57
7
bDefine set of …
Rules/Metrics related to reqts. Qualities
0
4
13
21
1.24
13
cDevelop KM tool …
Categorize reqts.; cost est.; risk prediction
0
4
11
19
1.27
6
dDefine a representation … Tools; Automated measurement & analysis
0
3
11
17
1.21
6
eExtend USC COCOMO … Reqts to SLOC
0
7
8
22
1.47
6
fDevelop an approach …
Costing by capabilities; reqts. definition by analogy
0
6
8
20
1.43
7
gDevelop meta-model, …
Meta-models, semantics & ontologies
0
8
6
22
1.57
7
hDefine the minimum …
Steps for lean requirements processes
4
9
2
32
2.13
7
iDefine a mechanism …
Test-centric reqts. development
0
12
1
25
1.92
6
jDevelop a model …
Predict reqts. volatility
0
3
13
19
1.19
14
35
Ease / Value Chart
16
14
j
b
Value (# in top 10)
12
10
8
f
6
d
c
a
g
h
i
e
4
2
0
1.00
1.20
1.40
1.60
1.80
2.00
2.20
Ease (1=Difficult)
36