Tradespace, Affordability, and
COCOMO III
Barry Boehm, USC
CSSE Annual Research Review 2014
April 30, 2014
03-19-2014
1
Tradespace, Affordability and COCOMO III
• Critical nature of the ilities
–
–
–
–
Major source of project overruns, failures
Significant source of stakeholder value conflicts
Poorly defined, understood
Underemphasized in project management
• SERC Foundations efforts
–
–
–
–
Stakeholder value-based, means-ends hierarchy
Formal analysis of ility definitions and relations
Architecture strategy synergies and conflicts
Affordability means-ends hierarchy
• Relations to COCOMO III
03-19-2014
2
Importance of ility Tradeoffs
Major source of DoD system overruns
• System ilities have systemwide impact
– System elements generally just have local impact
• ilities often exhibit asymptotic behavior
– Watch out for the knee of the curve
• Best architecture is a discontinuous function of ility level
– “Build it quickly, tune or fix it later” highly risky
– Large system example below
03-19-2014
3
Example of Current Practice
• “The system shall have a Mean Time Between Failures of
10,000 hours”
• What is a “failure?”
– 10,000 hours on liveness
– But several dropped or garbled messages per hour?
• What is the operational context?
– Base operations? Field operations? Conflict operations?
• Most management practices focused on functions
– Requirements, design reviews; traceability matrices; work
breakdown structures; data item descriptions; earned value
management
• What are the effects on other –ilities?
– Cost, schedule, performance, maintainability?
03-19-2014
4
USC: COCOMO II-Based Tradeoff Analysis
Better, Cheaper, Faster: Pick Any Two?
9
(RELY, MTBF (hours))
8
(VL, 1)
Cost ($M)
7
(L, 10)
6
5
(N, 300)
4
(H, 10K)
3
(VH, 300K)
•For 100-KSLOC set of features
•Can “pick all three” with 77-KSLOC set of features
2
1
-- Cost/Schedule/RELY:
“pick any two” points
0
0
10
20
30
40
50
Development Time (Months)
03-19-2014
5
Tradespace, Affordability and COCOMO III
• Critical nature of the ilities
–
–
–
–
Major source of project overruns, failures
Significant source of stakeholder value conflicts
Poorly defined, understood
Underemphasized in project management
• SERC Foundations efforts
–
–
–
–
Stakeholder value-based, means-ends hierarchy
Formal analysis of ility definitions and relations
Architecture strategy synergies and conflicts
Affordability means-ends hierarchy
• Relations to COCOMO III
03-19-2014
6
SERC Value-Based ilities Hierarchy
Based on ISO/IEC 9126, 25030; JCIDS; previous SERC research
• Individual ilities
– Mission Effectiveness: Speed, Physical Capability, Cyber Capability, Usability,
Accuracy, Impact, Endurability, Maneuverability, Scalability, Versatility
– Resource Utilization: Cost, Duration, Personnel, Scarce Quantities (capacity,
weight, energy, …); Manufacturability, Sustainability
– Protection: Security, Safety
– Robustness: Reliability, Availablilty, Maintainability, Survivability
– Flexibility: Modifiability, Tailorability, Adaptability
– Composability: Interoperability, Openness, Service-Orientation
• Composite ilities
–
–
–
–
Comprehensiveness/Suitability: all of the above
Dependability: Mission Effectiveness, Protection, Robustness
Resilience: Protection, Robustness, Flexibility
Affordability: Mission Effectiveness, Resource Utilization
03-19-2014
7
Means-Ends Framework: Affordability
Get the Best from People
Make Tasks More Efficient
Affordability
Improvements
and Tradeoffs
Eliminate Tasks
Eliminate Scrap, Rework
Simplify Products (KISS)
Staffing, Incentivizing, Teambuilding
Facilities, Support Services
Kaizen (continuous improvement)
Tools and Automation
Work and Oversight Streamlining
Collaboration Technology
Lean and Agile Methods
Task Automation
Model-Based Product Generation
Early Risk and Defect Elimination
Evidence-Based Decision Gates
Modularity Around Sources of Change
Incremental, Evolutionary Development
Value-Based, Agile Process Maturity
Risk-Based Prototyping
Value-Based Capability Prioritization
Satisficing vs. Optimizing Performance
Reuse Components
Domain Engineering and Architecture
Composable Components,Services, COTS
Legacy System Repurposing
Reduce Operations, Support Costs
Automate Operations Elements
Design for Maintainability, Evolvability
Streamline Supply Chain
Anticipate, Prepare for Change
Value- and Architecture-Based
Tradeoffs and Balancing
03-19-2014
8
Architecture Strategy Synergy-Conflict Matrix
03-19-2014
9
Software Development Cost vs. Quality
1.4
1.3
Relative
Cost to
Develop
1.26
1.2
1.1
1.10
1.0
1.0
0.9
0.8
0.92
0.82
Very
Low
Low
Nominal
High
COCOMO II RELY Rating
MTBF (hours)
03-19-2014
1
10
300
10,000
Very
High
300,000
10
Software Ownership Cost vs. Quality
1.4
VL = 2.55
L = 1.52
1.3
Relative
Cost to
Develop,
Maintain,
Own and
Operate
Operational-defect cost at Nominal dependability
= Software life cycle cost
1.26
1.2
1.23
1.20
1.1
1.0
1.10
1.10
1.05
1.07
1.11
Operational defect cost = 0
1.07
0.99
0.9
0.8
0.92
0.76
0.82
Very
Low
0.69
Low
Nominal
High
COCOMO II RELY Rating
MTBF (hours)
03-19-2014
70%
Maint.
1
10
300
10,000
Very
High
300,000
11
03-19-2014
12
Tradespace, Affordability and COCOMO III
• Critical nature of the ilities
–
–
–
–
Major source of project overruns, failures
Significant source of stakeholder value conflicts
Poorly defined, understood
Underemphasized in project management
• SERC Foundations efforts
–
–
–
–
Stakeholder value-based, means-ends hierarchy
Formal analysis of ility definitions and relations
Architecture strategy synergies and conflicts
Affordability means-ends hierarchy
• Relations to COCOMO III
03-19-2014
13
Affordability and Tradespace Framework
Staffing, Incentivizing, Teambuilding
Facilities, Support Services
Kaizen (continuous improvement)
Get the Best from People
Make Tasks More Efficient
Affordability
Improvements
and Tradeoffs
Tools and Automation
Work and Oversight Streamlining
Collaboration Technology
Lean and Agile Methods
Eliminate Tasks
Task Automation
Model-Based Product Generation
Early Risk and Defect Elimination
Evidence-Based Decision Gates
Modularity Around Sources of Change
Incremental, Evolutionary Development
Value-Based, Agile Process Maturity
Eliminate Scrap, Rework
Risk-Based Prototyping
Simplify Products (KISS)
Value-Based Capability Prioritization
Satisficing vs. Optimizing Performance
Reuse Components
Domain Engineering and Architecture
Composable Components,Services, COTS
Legacy System Repurposing
Reduce Operations, Support Costs
Value- and Architecture-Based
Tradeoffs and Balancing
14
03-19-2014
Automate Operations Elements
Design for Maintainability, Evolvability
Streamline Supply Chain
Anticipate, Prepare for Change
Costing Insights: COCOMO II Productivity Ranges
Scale Factor Ranges: 10, 100, 1000 KSLOC
Development Flexibility (FLEX)
Staffing
Team Cohesion (TEAM)
Develop for Reuse (RUSE)
Teambuilding
Precedentedness (PREC)
Continuous
Improvement
Architecture and Risk Resolution (RESL)
Platform Experience (PEXP)
Data Base Size (DATA)
Required Development Schedule (SCED)
Language and Tools Experience (LTEX)
Process Maturity (PMAT)
Storage Constraint (STOR)
Use of Software Tools (TOOL)
Platform Volatility (PVOL)
Applications Experience (AEXP)
Multi-Site Development (SITE)
Documentation Match to Life Cycle Needs (DOCU)
Required Software Reliability (RELY)
Personnel Continuity (PCON)
Time Constraint (TIME)
Programmer Capability (PCAP)
Analyst Capability (ACAP)
Product Complexity (CPLX)
1
1.2
1.4
1.6
1.8
2
2.2
Productivity Range
15
03-19-2014
2.4
COSYSMO Sys Engr Cost Drivers
Teambuilding
Continuous
Improvement
Staffing
03-19-2014
16
Tradespace and Affordability Framework
Staffing, Incentivizing, Teambuilding
Facilities, Support Services
Kaizen (continuous improvement)
Get the Best from People
Make Tasks More Efficient
Tools and Automation
Work and Oversight Streamlining
Collaboration Technology
Affordability
Improvements
and Tradeoffs
Lean and Agile Methods
Eliminate Tasks
Task Automation
Model-Based Product Generation
Early Risk and Defect Elimination
Evidence-Based Decision Gates
Modularity Around Sources of Change
Incremental, Evolutionary Development
Value-Based, Agile Process Maturity
Eliminate Scrap, Rework
Risk-Based Prototyping
Simplify Products (KISS)
Value-Based Capability Prioritization
Satisficing vs. Optimizing Performance
Reuse Components
Domain Engineering and Architecture
Composable Components,Services, COTS
Legacy System Repurposing
Reduce Operations, Support Costs
Automate Operations Elements
Design for Maintainability, Evolvability
Streamline Supply Chain
Anticipate, Prepare for Change
Value- and Architecture-Based
Tradeoffs and Balancing
17
03-19-2014
Value-Based Testing: Empirical Data and ROI
— LiGuo Huang, ISESE 2005
100
(a)
% of
Value
for
Correct
Customer
Billing
Bullock data
– Pareto distribution
80
60
Automated test
generation (ATG) tool
- all tests have equal value
40
20
5
10
15
Customer Type
(b)
Return On Investment (ROI)
2
1.5
1
0.5
0
0
10
20
30
40
50
60
70
80
-0.5
-1
-1.5
% Tests Run
Value-Neutral ATG Testing
03-19-2014
18
Value-Based Pareto Testing
90
100
Value-Neutral Defect Fixing Is Even Worse
Pareto 80-20 Business Value
100
Automated test
generation tool
- all tests have equal value
80
% of
Value
for
Correct
Customer
Billing
60
40
Value-neutral defect fixing:
Quickly reduce # of defects
20
5
10
15
Customer Type
03-19-2014
19
Product Line Engineering and
Management: NPS
02-25-14
20
SysML Building Blocks for Cost Modeling
GaTech-USC Work in RT46 Phase 2 (Oct-Dec 2013)
• Implemented reusable SysML building blocks
– Based on SoS/COSYSMO SE cost (effort)
modeling work by Lane, Valerdi, Boehm, et al.
• Successfully applied building blocks to
healthcare SoS case study from [Lane 2009]
• Provides key step towards affordability trade studies
involving diverse “-ilities” (see MIM slides)
21
Healthcare SoS Case Study [Lane 2009] Implemented
Using SysML Building Blocks: Selected SysML Diagrams
22
Tradespace, Affordability and COCOMO III
• Critical nature of the ilities
–
–
–
–
Major source of project overruns, failures
Significant source of stakeholder value conflicts
Poorly defined, understood
Underemphasized in project management
• SERC Foundations efforts
–
–
–
–
Stakeholder value-based, means-ends hierarchy
Formal analysis of ility definitions and relations
Architecture strategy synergies and conflicts
Affordability means-ends hierarchy
• Relations to COCOMO III
03-19-2014
23
COCOMO II Data by 5-Year Periods
1/13/2014
© USC-CSSE
24
COCOMO II Data: Productivity Trends
1/13/2014
© USC-CSSE
25
COCOMO II Data: Process Maturity Trends
1/13/2014
© USC-CSSE
26
SRDR Data: Productivity vs. Size, CMM Level
400
350
300
250
0-1 EKSLOC
1-10 EKSLOC
200
10-50 EKSLOC
50-100 EKSLOC
150
>100 EKSLOC
100
50
0
CMM 2
1/13/2014
CMM 3
CMM 4
CMM 5
CMMI 3
© USC-CSSE
CMMI 4
CMMI 5
27
Trends Confounded by Missing Variables
Incremental Development Productivity Decline
QMP
12
10
8
Productivity
6
Linear (Productivity)
Log. (Productivity)
4
y = -0.7989x + 8.5493
R² = 0.3693
2
y = -2.708ln(x) + 8.7233
R² = 0.5326
0
1
12/03/2013
2
3
4
5
Copyright © USC-CSSE
6
28
COCOMO II Status and COCOMO III Plans
• Baseline COCOMO II calibrated to 161 project data points
– Pred (20%; 30%) = (63%; 70%) general; (75%;80%) local
• Added 149 data points 2000-2009
– Pred (30%) < 40% general; some sources ~70-80% local
– Some improvement with added variables (year; domain; agility)
• But some data-source mismatches unexplainable
• Vu Nguyen analysis of full dataset suggests further adjusting
– Rating scales for experience, tools, reliability
• Proposed approach for COCOMO III
– Explore models for unexplained existing sources or drop
– Try added variables for mostly-general fit to existing data
– Obtain more data to validate results
1/13/2014
© USC-CSSE
29