ilities Tradespace Workshop Summary
Barry Boehm, Supannika Koolmanojwong
USC-CSSE ARR 20
March 14, 2013
03-12-2013
1
Importance of ility Tradeoffs
Major source of DoD system overruns
•
System ilities have systemwide impact
•
ilities often exhibit asymptotic behavior
•
Best architecture is a discontinuous function of ility level
–
–
–
–
03-12-2013
System elements generally just have local impact
Watch out for the knee of the curve
“Build it quickly, tune or fix it later” highly risky
Large system example below
2
Importance of Cyber-Physical Systems
Major gap in tradespace analysis capabilities
• Current
ERS, DARPA tradespace research focused
on physical system tradeoffs
– Range, payload, size, weight, lethality, power and fuel
consumption, communications bandwidth, etc.
– Some focus on physical modularity, composability
• Current cyber tradespace research focused on
software, computing, human factors tradeoffs
– security, safety, interoperability, usability, flexibility,
adaptability, dependability, response time,
throughput, etc.
• Gaps in capabilities for co-design of hardware and
software, integration of tradespace analyses
03-12-2013
3
iTAP Startup Results
• Top-level ility hierarchy view
– Relation to JCIDS combat command user view
– Draft survey to compare with acquirer, developer, supporter views
• Affordability means-ends framework view
– Quantification via COCOMO, COSYSMO, CORADMO models
• Architecture-strategy synergies and conflicts views
– Conflicts with other capabilities
• Change-Oriented Views: Incremental Commitment
– The Cones of Uncertainty
– MIT Epoch-Era Approach
– Agile SE Schedule Acceleration Model
• Domain-Oriented Views
– Ground: Wayne State, Georgia Tech
– Sea: NPS; Air: AFIT; Space: MIT, Stevens, USC, U. Virginia
03-12-2013
4
SERC Value-Based ilities Hierarchy
Based on ISO/IEC 9126, 25030; JCIDS; previous SERC research
• Individual ilities
– Quality of Service: Performance, Accuracy, Usability, Scalability, Versatility
– Resource Utilization: Cost, Duration, Personnel, Scarce Quantities (size,
weight, energy, …)
– Protection: Safety, Security, Privacy
– Robustness: Reliability, Availablilty, Maintainability
– Flexibility: Modifiability, Tailorability/Extendability, Adaptability
– Composability: Interoperability/Portability, Openness/Standards Compliance,
Service-Orientation
• Composite ilities
–
–
–
–
Comprehensiveness/Suitability: all of the above
Dependability: Quality of Service, Protection, Robustness
Resilience: Protection, Robustness, Flexibility
Affordability: Quality of Service, Resource Utilization
03-12-2013
5
Prioritized JCIDS ilities
User View by Combatant Commands: Top priority first
• Intelligence, Surveillance, and Reconnaissance
– Comprehensive Persistent Survivable Integrated Timely Credible Adaptable
Innovative
• Command and Control (note emphasis on Usability
aspects)
– Interoperability Understanding Timeliness Accessibility Simplicity
Completeness Agility Accuracy Relevance Robustness Operational Trust
• Logistics: Supply
– Responsiveness Sustainability Flexibility Survivability Attainability Economy
Simplicity
• Logistics: Maintenance
– Sustainability Responsiveness Attainability Flexibility Economy Survivability
Simplicity
• Net-Centric: Information Transport
– Accessible Capacity Accurate Timely Throughput Expeditionary Latency
03-12-2013
6
Means-Ends Framework: Affordability
Get the Best from People
Make Tasks More Efficient
Affordability
Improvements and
Tradeoffs
Eliminate Tasks
Eliminate Scrap, Rework
Staffing, Incentivizing, Teambuilding
Facilities, Support Services
Kaizen (continuous improvement)
Tools and Automation
Work and Oversight Streamlining
Collaboration Technology
Lean and Agile Methods
Task Automation
Model-Based Product Generation
Early Risk and Defect Elimination
Evidence-Based Decision Gates
Modularity Around Sources of Change
Incremental, Evolutionary Development
Value-Based, Agile Process Maturity
Simplify Products (KISS)
Risk-Based Prototyping
Value-Based Capability Prioritization
Satisficing vs. Optimizing Performance
Reuse Components
Domain Engineering and Architecture
Composable Components,Services, COTS
Legacy System Repurposing
Reduce Operations, Support Costs
Automate Operations Elements
Design for Maintainability, Evolvability
Streamline Supply Chain
Anticipate, Prepare for Change
Value- and Architecture-Based
Tradeoffs and Balancing
03-12-2013
7
USC: COCOMO II-Based Tradeoff Analysis
Better, Cheaper, Faster: Pick Any Two?
Slider-based equalizer version being developed
9
(RELY, MTBF (hours))
8
(VL, 1)
Cost ($M)
7
(L, 10)
6
5
(N, 300)
4
(H, 10K)
3
(VH, 300K)
•For 100-KSLOC set of features
•Can “pick all three” with 77-KSLOC set of features
2
1
-- Cost/Schedule/RELY:
“pick any two” points
0
0
10
20
30
40
50
Development Time (Months)
03-12-2013
8
iTAP
Startup
• Top-level ility
hierarchy
view
Results
– Relation to JCIDS combat command user view
– Draft survey to compare with acquirer, developer, supporter views
• Affordability means-ends framework view
– Quantification via COCOMO, COSYSMO, CORADMO models
• Architecture-strategy synergies and conflicts views
– Conflicts with other capabilities
• Change-Oriented Views: Incremental Commitment
– The Cones of Uncertainty
– MIT Epoch-Era Approach
– Agile SE Schedule Acceleration Model
• Domain-Oriented Views
– Ground: Wayne State, Georgia Tech
– Sea: NPS; Air: AFIT; Space: MIT, Stevens, USC, U. Virginia
03-12-2013
9
Architecture-Based Attribute Trades:
Flexibility Example (RT-18a)
Flexibility Arch. Strategy
Synergies
Conflicts
High module cohesion;
Low module coupling
Interoperability
Reliability
High Performance via
Tight coupling
Service-oriented architecture
Composability, Usability, Testability
High Performance via
Tight coupling
Autonomous adaptive systems
Affordability via task automation;
Response time
Excess autonomy reduces human
Controllability
Modularization around sources of
change
Interoperability, Usability,
Reliability, Availability
Extra time on critical path of Rapid
Fielding
Multi-layered architecture
Reliability, Availability
Lower Performance due to layer
traversal overhead
Many built-in options, entry points
Functionality, Accessibility
Reduced Usability via options
proliferation; harder to Secure
User programmability
Usability, Mission Effectiveness
Full programmability causes
Reliability, Safety, Security risks
Spare/expandable capacity
Performance, Reliability
Added cost
Product line architecture, reusable
components
Cost, Schedule, Reliability
Some loss of performance vs.
optimized stovepipes
03-12-2013
10
MIT: ilities in Tradespace Exploration
Based on Lean Aerospace, DARPA research
Changeability
Enabling Construct: Tradespace Networks
More changeable
(ie including flexible,
adaptable, scalable
and modifiable)
Colored by
outdegree
For this plot, Ĉ=C∞
Survivability
Enabling Construct: Epochs and Eras
Value Robustness
Set of Metrics
03-12-2013
11
WSU: Versatility Factors and Physical Organization
Components that Can be in Different Positions or Orientations
Isolated or Separated Compartments
Sight
Mass & Structure Properties
•Mass
•Angular moments
•Imbalances*
•Load bearing wall strength
•Deck surface area
•Interior volumes**
•Interior surface areas**
Weapon
drive
drive
Turret
drive
Chassis
suspension
Running Gear
*Angular moments of the CG about axes of rotation
** By crew station and compartment
03-12-2013
12
Workshop objectives and approach
• Workshop objectives
– Identify interested collaborators and data
– Identify user needs for better reasoning about ility
tradeoffs and affordability
– Identify improved approaches for cyber-physical
system co-design
• Workshop approach
– Ask participants about their current and likely future
challenges and research needs
• For ilities and their tradeoffs
• For cyber-physical co-design
– Prioritize research with respect to strength of need,
difficulty
03-12-2013
13
Workshop approach
• Ask participants about their current and likely
future challenges and research needs
– For -ilities and their tradeoffs
– For cyber-physical co-design
• Prioritize research with respect to strength of
need, difficulty
Participants
•
•
•
•
•
•
•
•
•
Shawn Rahmani
Gary Hafen
Winsor Brown
J.D. Baker
Ed Colbert
Thammanoon K.
Peter Suk
Rachchabhorn W.
Sue K.
•
•
•
•
•
•
•
•
Qi Li
Lori Vaughan
Qing Wang
Jing Du
Liming Zhu
Da Yang
Lee Osterweil
Barry Boehm
Current and likely future challenges and research needs for -ilities and their tradeoffs
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Simulation model to address performance, usability model, comparing -ilities
Define hierarchy, top factors
Quantification of value of each –ilities, matrices
Contribution of each –ilities (H/M/L. option1/2)
Prototype for a short life vs requirements in terms of ilitlities
Scale the tradeoff, scale up?
Affordability (prioritizing issues, not only low cost)
Inter-dependency of -ilities
Pair-wise comparison tool (lockheed)
Issue of subjective tradeoff study
Modeling and trade study, matrices of architecture and –ilities
Healthcare group (right distribution of data vs performance, privacy control/security)
Usability vs design
Parametric diagram
Agile architecture development – tradeoff, architecture your asset & product
How to specify requirements in a measurable way
Security, accessibility, QoS
Scalability – esp. diseconomy of scale, e.g. mobile network
Reliability - 5 9s system in banking sector
Macro view & micro view; complimentary process modeling (agent, task, resources) optimize the resources & others to optimize the choices, a
tool that guide resource allocation (TWINS)
Statistical mechanic to understand the process & relationship to COCOMO
How to deploy, data /design selection; tradeoff guideline / tool for system implementation
Architectural style tradeoff; compare & normalize & confidence level; validation tool &Sensitivity analysis tool to improve confidence level
Fault tolerance; process view supporting tradeoff analysis
How many (functional ) requirements is enough; size, # per release , customized development vs maintenance/enhancement
unstated non-functional requirements
Estimation – over/under-estimation
Adaptability & flexibility – adapt to emergent behavior/requirements – not enough representative users
How to help decision makers – can we perform as stated in the proposal
Cyber vs physical; dual cone of uncertainty;
Guidance /model ; how to do the trade/ how to weigh, checklist ; human process part
Process for continuous delivery, how to prioritize requirements and assign them into each release based on revenue (considering dependency
& process adoption)
Using checklist – definition of Done(value, cost, quality)
Scenario generators
Product vs program characteristics - executability
Prioritization result
Tools
-ilities
•Models & Simulations (21)
•Multilevel
•Architecting (17)
•Prioritization (12)
•Metrics/ Matrices (8)
•Scenario generators (8)
•Checklists (7)
•Parametric diagrams (2)
•Affordability (14)
•Timeliness (12)
•Reliability / Fault Tolerance (11)
•Safety
•Scalability (10)
•Security / Privacy (10)
•Performance (5)
•Usability (4)
•Accessibility (3)
•Adaptability (3)
•Flexibility (1)
Afternoon Agenda
• Exploring high score -ilities factors
– Exploring degree of difficulties
• Exploring high score tools
• Exploring cyber-physical co-design methods
Affordability
•
•
•
•
Definition = Effectiveness & Cost (INCOSE)
Fixed effectiveness & try to reduce cost
Define cost (Total cost of ownership, life cycle cost)
Cost = including operational & support cost? System
cost (e.g. cheap to build to maintain but not cheap to
operate)
• What is the biggest gap that is not covered when doing
affordability analysis?
– Technical debt
• As long as you are aware of your debt, give you leeway
• Technical debt identification tool
– HW SW integration
Timeliness
• Define timeliness
– Meet deadline subject to definition of effectiveness (timebox – discard
low priority items to meet schedule; agile rebaselining)
– Version control issues
– Conflict in reusing items
Reliability / Fault Tolerance /Safety
•
•
•
•
Standard assurance issue
Define “safe”
Reliability = impact of defect (loss of life/properties/ $)
Dependent on other systems
– Cloud – no full control
•
•
•
•
•
•
•
How can your system tolerate other systems?
Not equal (reliable <> fault tolerance <> safety)
Acceptable levels
A tool to provide analysis of balancing these factors
KPP – key performance parameter – no single number
Estimating certification cost & schedule
Level of testing
Scalability
• Scalability of Product
• Architecture evaluation tool
– Something like static source code analysis tool
•
•
•
•
•
Modeling tool
Network traffic & overhead
Communication mechanism
Co-dependent systems (with internet connection? )
Diversity
– Versions or platforms
•
•
•
•
Horizontal vs Vertical (scale out vs scale up)
Consistency among diversity/nodes
Timing
How to upgrade
Security / Privacy
•
•
•
•
•
•
•
•
•
•
•
•
•
Acceptable levels / level of assurance
Security vs complexity tradeoff (also with reliability, availability)
multiple independent level of security
Certification cost & schedule
Scope of certification (privacy)
Lifetime of data (dynamic data)
Anti-tamper
communication security
Scaling
How to show feasibility evidence
How to justify being assured (how the entire thing is developed)
Proof of correctness
Domain dependent
Prioritization result
Tools
-ilities
•Models & Simulations (21)
•Multilevel
•Architecting (17)
•Prioritization (12)
•Metrics/ Matrices (8)
•Scenario generators (8)
•Checklists (7)
•Parametric diagrams (2)
•Affordability (14)
•Timeliness (12)
•Reliability / Fault Tolerance (11)
•Safety
•Scalability (10)
•Security / Privacy (10)
•Performance (5)
•Usability (4)
•Accessibility (3)
•Adaptability (3)
•Flexibility (1)
Models & Simulations (Multilevel)
•
•
•
•
•
•
•
Accuracy / fidelity
VV&A (verification, Validation & accreditation)
Scalability
Modeling environment (jungle/desert)
Modeling language
Visualization
Test cases, usage(scope of use), underlying
assumption
Architecting
•
•
•
•
•
•
•
Language (dependent)
Visualization (static & dynamic)
Executable architecture
Analyzable
Multiple views (like DODAF), integration of views
Generation of the system from the architecture
Architecture pattern, reuse
Prioritization
• Multiple stakeholder value proposition,
criteria analysis
• Tradeoff
• Interdependencies
• Cost & schedule
• Value estimation
• Environment (scenario)