Balancing System and Software Qualities
Barry Boehm, USC
STC 2015 Tutorial
October 12, 2015
Fall 2015
1
Outline
• Critical nature of system qualities (SQs)
–
–
–
–
–
Or non-functional requirements (NFRs); ilities
Major source of project overruns, failures
Significant source of stakeholder value conflicts
Poorly defined, understood
Underemphasized in project management
• Foundations: An initial SQs ontology
– Nature of an ontology; choice of IDEF5 structure
– Stakeholder value-based, means-ends hierarchy
– Synergies and Conflicts matrix and expansions
• SQ Opportunity Trees
• Conclusions
Fall 2015
2
Importance of SQ Tradeoffs
Major source of DoD, other system overruns
• SQs have systemwide impact
– System elements generally just have local impact
• SQs often exhibit asymptotic behavior
– Watch out for the knee of the curve
• Best architecture is a discontinuous function of SQ level
– “Build it quickly, tune or fix it later” highly risky
– Large system example below
$100M
$50M
Required
Architecture:
Custom; many
cache processors
Original Cost
Original Spec
1
Fall 2015
Original
Architecture:
Modified
Client-Server
3
2
Response Time (sec)
After
Prototyping
4
5
3
Types of Decision Reviews
• Schedule-based commitment reviews (plan-driven)
– We’ll release the RFP on April 1 based on the schedule in the plan
– $70M overrun to produce overperforming system
• Event-based commitment reviews (artifact-driven)
– The design will be done in 15 months, so we’ll have the review then
– Responsive design found to be unaffordable 15 months later
• Evidence-based commitment reviews (risk-driven)
– Evidence: affordable COTS-based system can’t satisfy 1-second
requirement
• Custom solution roughly 3x more expensive
– Need to reconsider 1-second requirement
Fall 2015
4
Evidence-Based Decision Result
•
Attempt to validate 1-second response time
• Commercial system benchmarking and architecture
analysis: needs expensive custom solution
• Prototype: 4-second response time OK 90% of the time
• Negotiate response time ranges
• 2 seconds desirable
• 4 seconds acceptable with some 2-second special cases
• Benchmark commercial system add-ons to validate
their feasibility
• Present solution and feasibility evidence at evidencebased decision review
• Result: Acceptable solution with minimal delay
Fall 2015
5
Outline
• Critical nature of system qualities (SQs)
–
–
–
–
–
Or non-functional requirements (NFRs); ilities
Major source of project overruns, failures
Significant source of stakeholder value conflicts
Poorly defined, understood
Underemphasized in project management
• Foundations: An initial SQs ontology
– Nature of an ontology; choice of IDEF5 structure
– Stakeholder value-based, means-ends hierarchy
– Synergies and Conflicts matrix and expansions
• SQ Opportunity Trees
• Conclusions
Fall 2015
6
Example of SQ Value Conflicts: Security IPT
• Single-agent key distribution; single data copy
– Reliability: single points of failure
• Elaborate multilayer defense
– Performance: 50% overhead; real-time deadline problems
• Elaborate authentication
– Usability: delays, delegation problems; GUI complexity
• Everything at highest level
– Modifiability: overly complex changes, recertification
Fall 2015
7
Proliferation of Definitions: Resilience
• Wikipedia Resilience variants: Climate, Ecology, Energy Development,
Engineering and Construction, Network, Organizational, Psychological, Soil
• Ecology and Society Organization Resilience variants: Original-ecological,
Extended-ecological, Walker et al. list, Folke et al. list; Systemic-heuristic,
Operational, Sociological, Ecological-economic, Social-ecological system,
Metaphoric, Sustainabilty-related
• Variants in resilience outcomes
– Returning to original state; Restoring or improving original state;
Maintaining same relationships among state variables; Maintaining
desired services; Maintaining an acceptable level of service; Retaining
essentially the same function, structure, and feedbacks; Absorbing
disturbances; Coping with disturbances; Self-organizing; Learning and
adaptation; Creating lasting value
– Source of serious cross-discipline collaboration problems
Fall 2015
8
Example of Current Practice
• “The system shall have a Mean Time Between Failures of
10,000 hours”
• What is a “failure?”
– 10,000 hours on liveness
– But several dropped or garbled messages per hour?
• What is the operational context?
– Base operations? Field operations? Conflict operations?
• Most management practices focused on functions
– Requirements, design reviews; traceability matrices; work
breakdown structures; data item descriptions; earned value
management
• What are the effects of or on other SQs?
– Cost, schedule, performance, maintainability?
Fall 2015
9
Outline
• Critical nature of system qualities (SQs)
–
–
–
–
–
Or non-functional requirements (NFRs); ilities
Major source of project overruns, failures
Significant source of stakeholder value conflicts
Poorly defined, understood
Underemphasized in project management
• Foundations: An initial SQs ontology
– Nature of an ontology; choice of IDEF5 structure
– Stakeholder value-based, means-ends hierarchy
– Synergies and Conflicts matrix and expansions
• SQ Opportunity Trees
• Conclusions
Fall 2015
10
Need for SQs Ontology
• Oversimplified one-size-fits all definitions
– ISO/IEC 25010, Reliability: the degree to which a system ,
product, or component performs specified functions under
specified conditions for a specified period of time
– OK if specifications are precise, but increasingly “specified
conditions” are informal, sunny-day user stories.
• Satisfying just these will pass “ISO/IEC Reliability,” even if system
fails on rainy-day user stories
– Need to reflect that different stakeholders rely on different
capabilities (functions, performance, flexibility, etc.) at
different times and in different environments
• Proliferation of definitions, as with Resilience
• Weak understanding of inter-SQ relationships
– Security Synergies and Conflicts with other qualities
Fall 2015
11
Nature of an ontology; choice of IDEF5 structure
• An ontology for a collection of elements is a definition of
what it means to be a member of the collection
• For “system qualities,” this means that an SQ identifies an
aspect of “how well” the system performs
– The ontology also identifies the sources of variability in the
value of “how well” the system performs
• Functional requirements specify “what;” NFRs specify “how well”
• After investigating several ontology frameworks, the IDEF5
framework appeared to best address the nature and sources
of variability of system SQs
– Good fit so far
Fall 2015
12
Initial SERC SQs Ontology
• Modified version of IDEF5 ontology framework
– Classes, Subclasses, and Individuals
– Referents, States, Processes, and Relations
• Top classes cover stakeholder value propositions
– Mission Effectiveness, Resource Utilization, Dependability, Changeabiity
• Subclasses identify means for achieving higher-class ends
– Means-ends one-to-many for top classes
– Ideally mutually exclusive and exhaustive, but some exceptions
– Many-to-many for lower-level subclasses
• Referents, States, Processes, Relations cover SQ variation
•
•
•
•
Referents: Sources of variation by stakeholder value context:
States: Internal (beta-test); External (rural, temperate, sunny)
Processes: Operational scenarios (normal vs. crisis; experts vs. novices)
Relations: Impact of other SQs (security as above, synergies & conflicts)
Fall 2015
13
Example: Reliability Revisited
• Reliability is the probability that the system will deliver
stakeholder-satisfactory results for a given time period
(generally an hour), given specified ranges of:
– Stakeholder value propositions: desired and acceptable
ranges of liveness, accuracy, response time, speed,
capabilities, etc.
– System internal and external states: integration test,
acceptance test, field test, etc.; weather, terrain, DEFCON,
takeoff/flight/landing, etc.
– System internal and external processes: status checking
frequency; types of missions supported; workload volume,
interoperability with independently evolving external systems
– Effects via relations with other SQs: synergies improving other
SQs; conflicts degrading other SQs
Fall 2015
14
Set-Based SQs Definition Convergence
RPV Surveillance Example
Phase 1
Desired
Phase 2
Effective
ness
(pixels/
frame)
Phase 3
Acceptable
Accept
able
Desired
Efficiency (E.g., frames/second)
Phase 1. Rough ConOps, Rqts, Solution Understanding
Phase 2. Improved ConOps, Rqts, Solution Understanding
Phase 3. Good ConOps, Rqts, Solution Understanding
Fall 2015
15
Stakeholder value-based, means-ends hierarchy
• Mission operators and managers want improved Mission Effectiveness
– Involves Physical Capability, Cyber Capability, Human Usability, Speed, Accuracy,
Impact, Maneuverability, Scalability, Versatility, Interoperability
• Mission investors and system owners want Mission Cost-Effectiveness
– Involves Cost, Duration, Personnel, Scarce Quantities (capacity, weight, energy, …);
Manufacturability, Sustainability
• All want system Dependability: cost-effective defect-freedom, availability, and
safety and security for the communities that they serve
– Involves Reliability, Availability, Maintainability, Survivability, Safety, Security,
Robustness
• In an increasingly dynamic world, all want system Changeability: to be rapidly
and cost-effectively evolvable
– Involves Maintainability, Adaptability
Fall 2015
16
U. Virginia: Coq Formal Reasoning Structure
• Inductive Dependable (s: System): Prop :=
•
mk_dependability: Security s -> Safety s -> Reliability s ->
•
Maintainability s -> Availability s -> Survivability s ->
•
Robustness s -> Dependable s.
•
•
•
•
•
•
•
•
•
•
Example aSystemisDependable: Dependable aSystem.
apply mk_dependability.
exact (is_secure aSystem).
exact (is_safe aSystem).
exact (is_reliable aSystem).
exact (is_maintainable aSystem).
exact (is_avaliable aSystem).
exact (is_survivable aSystem).
exact (is_robust aSystem).
Qed.
Fall 2015
17
Maintainability Challenges and Responses
•
Maintainability supports both Dependability and Changeability
– Distinguish between Repairability and Modifiability
– Elaborate each via means-ends relationships
•
Multiple definitions of Changeability
– Distinguish between Product Quality and Quality in Use (ISO/IEC 25010)
– Provide mapping between Product Quality and Quality in Use viewpoints
•
Changeability can be both external and internal
– Distinguish between Maintainability and Adaptability
•
Many definitions of Resilience
– Define Resilience as a combination of Dependability and Changeability
•
Variability of Dependability and Changeability values
– Ontology addresses sources of variation
– Referents (stakeholder values), States, Processes, Relations with other SQs
•
Need to help stakeholders choose options, avoid pitfalls
– Qualipedia, Synergies and Conflicts, Opportunity Trees
Fall 2015
18
Product Quality View of Changeability
MIT Quality in Use View also valuable
• Changeability (PQ): Ability to become different product
– Swiss Army Knife, Brick Useful but not Changeable
• Changeability (Q in Use): Ability to accommodate changes in use
– Swiss Army Knife does not change as a product but is Changeable
– Brick is Changeable in Use (Can help build a wall or break a window?)
Changeability
Adaptability
Internal
• Self-Diagnosability
• Self-Modifiability
• Self-Testability
Subclass of
Means to End
Fall 2015
External
Maintainability
Modifiability
•
•
•
•
Understandability
Modularity
Scalability
Portability
Changes
Defects
Repairability
Testability
•
•
•
Diagnosability
Accessibility
Restorability
19
Dependability, Changeability, and Resilience
Resilience
Changeability
Dependability,
Availability
Reliability
Maintainability
Defect Freedom
Survivability
Fault Tolerance
Complete
Repairability
Modifiability
Testability
Test Plans, Coverage
Test Scenarios, Data
Test Drivers, Oracles
Test Software Qualitities
Partial
A
Robustness
Self-Repairability
Graceful
Degradation
Testability, Diagnosability, etc.
Fall 2015
Choices of
Security,
Safety
…
B
Means to End
A
B
C,D
Subclass of
20
Outline
• Critical nature of system qualities (SQs)
–
–
–
–
–
Or non-functional requirements; ilities
Major source of project overruns, failures
Significant source of stakeholder value conflicts
Poorly defined, understood
Underemphasized in project management
• Need for and nature of SQs ontology
– Nature of an ontology; choice of IDEF5 structure
– Stakeholder value-based, means-ends hierarchy
– Synergies and Conflicts matrix and expansions
• SQ Opportunity Trees
• Conclusions
Fall 2015
21
7x7 Synergies and Conflicts Matrix
• Mission Effectiveness expanded to 4 elements
– Physical Capability, Cyber Capability, Interoperability, Other
Mission Effectiveness (including Usability as Human Capability)
• Synergies and Conflicts among the 7 resulting elements
identified in 7x7 matrix
– Synergies above main diagonal, Conflicts below
• Work-in-progress tool will enable clicking on an entry and
obtaining details about the synergy or conflict
– Ideally quantitative; some examples next
• Still need synergies and conflicts within elements
– Example 3x3 Dependability subset provided
Fall 2015
22
Fall 2015
23
Software Development Cost vs. Reliability
1.4
1.3
Relative
Cost to
Develop
1.26
1.2
1.1
1.10
1.0
1.0
0.9
0.8
0.92
0.82
Very
Low
Low
Nominal
High
COCOMO II RELY Rating
MTBF (hours)
Fall 2015
1
10
300
10,000
Very
High
300,000
24
Software Ownership Cost vs. Reliability
1.4
VL = 2.55
L = 1.52
1.3
Relative
Cost to
Develop,
Maintain,
Own and
Operate
Operational-defect cost at Nominal dependability
= Software life cycle cost
1.26
1.2
1.23
1.20
1.1
1.0
1.10
1.10
1.05
1.07
1.11
Operational defect cost = 0
1.07
0.99
0.9
0.8
0.92
0.76
0.82
Very
Low
0.69
Low
Nominal
High
COCOMO II RELY Rating
MTBF (hours)
Fall 2015
70%
Maint.
1
10
300
10,000
Very
High
300,000
25
Affordability and Tradespace Framework
Staffing, Incentivizing, Teambuilding
Facilities, Support Services
Kaizen (continuous improvement)
Get the Best from People
Tools and Automation
Work and Oversight Streamlining
Collaboration Technology
Make Tasks More Efficient
Cost
Improvements
and Tradeoffs
Lean and Agile Methods
Eliminate Tasks
Task Automation
Model-Based Product Generation
Early Risk and Defect Elimination
Evidence-Based Decision Gates
Modularity Around Sources of Change
Incremental, Evolutionary Development
Value-Based, Agile Process Maturity
Eliminate Scrap, Rework
Risk-Based Prototyping
Simplify Products (KISS)
Value-Based Capability Prioritization
Satisficing vs. Optimizing Performance
Reuse Components
Domain Engineering and Architecture
Composable Components,Services, COTS
Legacy System Repurposing
Reduce Operations, Support Costs
Value- and Architecture-Based
Tradeoffs and Balancing
Fall 2015
Automate Operations Elements
Design for Maintainability, Evolvability
Streamline Supply Chain
Anticipate, Prepare for Change
26
Costing Insights: COCOMO II Productivity Ranges
Scale Factor Ranges: 10, 100, 1000 KSLOC
Development Flexibility (FLEX)
Staffing
Team Cohesion (TEAM)
Develop for Reuse (RUSE)
Teambuilding
Precedentedness (PREC)
Continuous
Improvement
Architecture and Risk Resolution (RESL)
Platform Experience (PEXP)
Data Base Size (DATA)
Required Development Schedule (SCED)
Language and Tools Experience (LTEX)
Process Maturity (PMAT)
Storage Constraint (STOR)
Use of Software Tools (TOOL)
Platform Volatility (PVOL)
Applications Experience (AEXP)
Multi-Site Development (SITE)
Documentation Match to Life Cycle Needs (DOCU)
Required Software Reliability (RELY)
Personnel Continuity (PCON)
Time Constraint (TIME)
Programmer Capability (PCAP)
Analyst Capability (ACAP)
Product Complexity (CPLX)
1
1.2
1.4
1.6
1.8
2
2.2
2.4
Productivity Range
Fall 2015
27
COSYSMO Sys Engr Cost Drivers
Teambuilding
Continuous
Improvement
Staffing
Fall 2015
28
Tradespace and Affordability Framework
Staffing, Incentivizing, Teambuilding
Facilities, Support Services
Kaizen (continuous improvement)
Get the Best from People
Tools and Automation
Work and Oversight Streamlining
Make Tasks More Efficient
Collaboration Technology
Affordability
Improvements
and Tradeoffs
Lean and Agile Methods
Eliminate Tasks
Task Automation
Model-Based Product Generation
Early Risk and Defect Elimination
Evidence-Based Decision Gates
Modularity Around Sources of Change
Incremental, Evolutionary Development
Value-Based, Agile Process Maturity
Eliminate Scrap, Rework
Risk-Based Prototyping
Simplify Products (KISS)
Value-Based Capability Prioritization
Satisficing vs. Optimizing Performance
Reuse Components
Domain Engineering and Architecture
Composable Components,Services, COTS
Legacy System Repurposing
Reduce Operations, Support Costs
Automate Operations Elements
Design for Maintainability, Evolvability
Streamline Supply Chain
Anticipate, Prepare for Change
Value- and Architecture-Based
Tradeoffs and Balancing
Fall 2015
29
COCOMO II-Based Tradeoff Analysis
Better, Cheaper, Faster: Pick Any Two?
9
(RELY, MTBF (hours))
8
(VL, 1)
Cost ($M)
7
(L, 10)
6
5
(N, 300)
4
(H, 10K)
3
•For 100-KSLOC set of features
•Can “pick all three” with 77-KSLOC set of features
2
(VH, 300K)
1
-- Cost/Schedule/RELY:
“pick any two” points
0
0
10
20
30
40
50
Development Time (Months)
Fall 2015
30
Value-Based Testing: Empirical Data and ROI
— LiGuo Huang, ISESE 2005
100
(a)
% of
Value
for
Correct
Customer
Billing
Bullock data
– Pareto distribution
80
60
Automated test
generation (ATG) tool
- all tests have equal value
40
20
5
10
15
Customer Type
(b)
Return On Investment (ROI)
2
1.5
1
0.5
0
0
10
20
30
40
50
60
70
80
90
100
-0.5
-1
-1.5
% Tests Run
Value-Neutral ATG Testing
Fall 2015
Value-Based Pareto Testing
31
Value-Neutral Defect Fixing Is Even Worse
Pareto 80-20 Business Value
100
Automated test
generation tool
- all tests have equal value
80
% of
Value
for
Correct
Customer
Billing
60
40
Value-neutral defect fixing:
Quickly reduce # of defects
20
5
10
15
Customer Type
Fall 2015
32
Outline
• Foundations: An initial SQs ontology
– Nature of an ontology; choice of IDEF5 structure
– Stakeholder value-based, means-ends hierarchy
• Critical nature of system qualities (SQs)
–
–
–
–
–
Or non-functional requirements (NFRs); ilities
Major source of project overruns, failures
Significant source of stakeholder value conflicts
Poorly defined, understood
Underemphasized in project management
• SQ Opportunity Trees
• Conclusions
Fall 2015
33
Context: SERC SQ Initiative Elements
• SQ Tradespace and Affordability Project foundations
–
–
–
–
More precise SQ definitions and relationships
Stakeholder value-based, means-ends relationships
SQ strategy effects, synergies, conflicts, Opportunity Trees
USC, MIT, U. Virginia
• Next-generation system cost-schedule estimation models
– Software: COCOMO III
– Systems Engineering: COSYSMO 3.0
– USC, AFIT, GaTech, NPS
• Applied SQ methods, processes, and tools (MPTs)
– For concurrent cyber-physical-human systems
– Experimental MPT piloting, evolution, improvement
– Wayne State, AFIT, GaTech, MIT, NPS, Penn State, USC
Fall 2015
34
SQ Opportunity Trees
• Means-Ends relationships for major SQs
– Previously developed for cost and schedule reduction
– Revising an early version for Dependability
– Starting on Maintainability
• Accommodates SQ Synergies and Conflicts matrix
– Conflicts via “Address Potential Conflicts” entry
– Means to Ends constitute Synergies
• More promising organizational structure than matrix
Fall 2015
35
Schedule Reduction Opportunity Tree
Business process reengineering
Reusing assets
Applications generation
Schedule as independent variable
Eliminating Tasks
Work streamlining (80-20)
Tools and automation
Increasing parallelism
Reducing Time Per Task
Reducing Risks of Single-Point
Failures
Reducing failures
Reducing their effects
Early error elimination
Process anchor points
Improving process maturity
Collaboration technology
Reducing Backtracking
Minimizing task dependencies
Avoiding high fan-in, fan-out
Reducing task variance
Removing tasks from critical path
24x7 development
Nightly builds, testing
Weekend warriors
Activity Network Streamlining
Increasing Effective Workweek
Better People and Incentives
Transition to Learning Organization
36
Fall 2015
Reuse at HP’s Queensferry
Telecommunication Division
70
Time
to
Market
60
(months)
40
Non-reuse Project
Reuse project
50
30
20
10
0
86
87
88
89
Year
37
Fall 2015
90
91
92
Software Dependability Opportunity Tree
Decrease
Defect
Prob (Loss)
Decrease
Defect
Risk
Exposure
Defect Prevention
Defect Detection
and Removal
Value/Risk - Based
Defect Reduction
Decrease
Defect
Impact,
Size (Loss)
Graceful Degradation
CI Methods and Metrics
Continuous
Improvement
Process, Product, People
Technology
38
Fall 2015
Software Defect Prevention Opportunity Tree
Defect
Prevention
People
practices
Standards
Languages
Prototyping
Modeling & Simulation
Reuse
Root cause analysis
Fall 2015
IPT, JAD, WinWin,…
PSP, Cleanroom, Pair development,…
Manual execution, scenarios,...
Staffing for dependability
Rqts., Design, Code,…
Interfaces, traceability,…
Checklists
39
Defect Detection Opportunity Tree
Automated
Analysis
Completeness checking
Consistency checking
- Views, interfaces, behavior,
pre/post conditions
Traceability checking
Defect
Detection
and Repair
- Rqts.
- Design
- Development
Compliance checking
- Models, assertions, standards
Peer reviews, inspections
Reviewing
Architecture Review Boards
Pair development
Requirements & design
Structural
Testing
Operational profile
Usage (alpha, beta)
Regression
Value/Risk - based
Test automation
Fall 2015
40
Defect Impact Reduction Opportunity Tree
Business case analysis
Value/Risk Based Defect
Reduction
Decrease
Defect
Impact,
Size (Loss)
Pareto (80-20) analysis
V/R-based reviews
V/R-based testing
Cost/schedule/quality
as independent variable
Fault tolerance
Graceful
Degredation
Self-stabilizing SW
Reduced-capability models
Manual Backup
Rapid recovery
Fall 2015
41
Repairability Opportunity Tree
Anticipate Repairability Needs
Repair facility requirements
Repair-type trend analysis
Repair skills analysis
Repair personnel involvement
Address Potential Conflicts
Design/Develop for Repairability
Repair facility design, development
Replacement parts trend analysis
Replacement parts logistics development
Replacement accessibility deaign
Address Potential Conflicts
Smart system anomaly, trend analysis
Informative error messages
Multimedia diagnosis guides
Fault localization
Switchable spare components
Improve Repair Diagnosis
Address Potential Conflicts
Improve Repair, V&V Processes
Prioritize, Schedule Repairs, V&V
Repair compatibility analysis
Regression test capabilities
Address Potential Conflicts
42
Fall 2015
Modifiability Opportunity Tree
Anticipate Modifiability Needs
Evolution requirements
Trend analysis
Hotspot (change source) analysis
Modifier involvement
Address Potential Conflicts
Design/Develop for Modifiability
Modularize around hotspots
Service-orientation; loose coupling
Spare capacity
Domain-specific architecture in domain
Enable user programmability
Address Potential Conflicts
Prioritize, Schedule Modifications, V&V
Modification compatibility analysis
Regression test capabilities
Improve Modification, V&V
Address Potential Conflicts
43
Fall 2015
Hotspot Analysis: Projects A and B
- Change processing over 1 person-month = 152 person-hours
Category
Project A
Project B
Extra long messages
3404+626+443+328+244= 5045
Network failover
2050+470+360+160= 3040
Hardware-software interface
620+200=
820
Encryption algorithms
1629+513+289+232+166= 2832
1247+368= 1615
Subcontractor interface
1100+760+200= 2060
GUI revision
980+730+420+240+180 =2550
Data compression algorithm
External applications interface
COTS upgrades
910
770+330+200+160= 1460
540+380+190= 1110
Database restructure
741+302+221+197= 1461
690+480+310+210+170= 1860
Routing algorithms
494+198= 692
Diagnostic aids
360
Fall 2015
477+318+184= 979
44
TOTAL:
13620
13531
Product Line Engineering and
Management: NPS
Fall 2015
45
Outline
• Critical nature of system qualities (SQs)
–
–
–
–
–
Or non-functional requirements (NFRs); ilities
Major source of project overruns, failures
Significant source of stakeholder value conflicts
Poorly defined, understood
Underemphasized in project management
• Foundations: An initial SQs ontology
– Nature of an ontology; choice of IDEF5 structure
– Stakeholder value-based, means-ends hierarchy
– Synergies and Conflicts matrix and expansions
• SQ Opportunity Trees
• Conclusions
Fall 2015
46
Conclusions
• System qualities (SQs) are success-critical
– Major source of project overruns, failures
– Significant source of stakeholder value conflicts
– Poorly defined, understood
– Underemphasized in project management
• SQs ontology clarifies nature of system qualities
– Using value-based, means-ends hierarchy
– Identifies variation types: referents, states, processes, relations
– Relations enable SQ synergies and conflicts identification
Fall 2015
47
List of Acronyms
CD
CP
DCR
DoD
ECR
EV
FCR
FED
GAO
Fall 2015
Concept Development
Competitive Prototyping
Development Commitment
Review
Department of Defense
Exploration Commitment
Review
Expected Value
Foundations Commitment
Review
Feasibility Evidence
Description
Government Accounting
Office
©USC-CSSE
ICM
KPP
MBASE
OCR
RE
RUP
V&V
VB
VCR
Incremental Commitment
Model
Key Performance Parameter
Model-Based Architecting
and Software Engineering
Operations Commitment
Review
Risk Exposure
Rational Unified Process
Verification and Validation
Value of Bold approach
Valuation Commitment
Review
48
References
B. Boehm, J. Lane, S. Koolmanojwong, R. Turner, The Incremental Commitment
Spiral Model, Addison Wesley, 2014.
B. Boehm, “Anchoring the Software Process,” IEEE Software, July 1996.
B. Boehm, A.W. Brown, V. Basili, and R. Turner, “Spiral Acquisition of SoftwareIntensive Systems of Systems,” Cross Talk, May 2004, pp. 4-9.
B. Boehm and J. Lane, “Using the ICM to Integrate System Acquisition,
Systems Engineering, and Software Engineering,” CrossTalk, October 2007, pp.
4-9.
J. Maranzano et. al., “Architecture Reviews: Practice and Experience,” IEEE
Software, March/April 2005.
R. Pew and A. Mavor, Human-System Integration in the System Development
Process: A New Look, National Academy Press, 2007.
W. Royce, Software Project Management, Addison Wesley, 1998.
R. Valerdi, “The Constructive Systems Engineering Cost Model,” Ph.D.
dissertation, USC, August 2005.
CrossTalk articles: www.stsc.hill.af.mil/crosstalk
Fall 2015
49
Backup charts
Fall 2015
50
Fall 2015
51
Metrics Evaluation Criteria
•
•
•
•
•
•
Correlation with quality: How much does the metric relate with our notion of
software quality? It implies that nearly all programs with a similar value of the
metric will possess a similar level of quality. This is a subjective correlational
measure, based on our experience.I
Importance: How important is the metric and are low or high values preferable
when measuring them? The scales, in descending order of priority are: Extremely
Important, Important and Good to have
Feasibility of automated evaluation: Are things fully or partially automable and
what kinds of metrics are obtainable?
Ease of automated evaluation: In case of automation how easy is it to compute
the metric? Does it involve mammoth effort to set up or can it be plug-and-play or
does it need to be developed from scratch? Any OTS tools readily available?
Completeness of automated evaluation: Does the automation completely capture
the metric value or is it inconclusive, requiring manual intervention? Do we need
to verify things manually or can we directly rely on the metric reported by the
tool?
Units: What units/measures are we using to quantify the metric?
Fall 2015
53