University of Southern California
Center for Systems and Software Engineering
Process and Product Architectures and Practices
for Achieving both Agility and High Assurance
----Tutorial----
Barry Boehm and Jo Ann Lane
University of Southern California
Center for Systems and Software Engineering
http://csse.usc.edu (tech report 2009-500)
Presented at
ICSE 2009, May 19, 2009
University of Southern California
Center for Systems and Software Engineering
Goals of Tutorial
• Describe
– Current and new challenges with respect to the
development of software-intensive systems
– How the Incremental Commitment Model (ICM) and its
capabilities can help developers meet these challenges
– Use of risk patterns to determine which system elements
can be done in an agile fashion and which should use
more formal processes
• Review several case studies that illustrate challenges and the
power of the ICM process decision table
• Conduct exercise to develop ICM special case model
May 19, 2009
Copyright © USC-CSSE
2
University of Southern California
Center for Systems and Software Engineering
Tutorial Outline
• Challenges for developing nextgeneration software systems
•
•
•
•
•
Overview of ICM
Risk-based balance of agility and assurance
ICM process decision table
Guidance and examples for using the ICM
ICM tailoring exercise
May 19, 2009
Copyright © USC-CSSE
3
University of Southern California
Center for Systems and Software Engineering
Future Process Challenges-I
• Multi-owner, multi-mission systems of systems (SoS)
– Integrated supply chain: strategic planning, marketing,
merchandising, outsourcing, just-in-time manufacturing,
logistics, finance, customer relations management
– Over 50 separately evolving external systems or services
– Need to satisfice among multiple stakeholders
– No one-size-fits-all solutions or processes
• Emergence and human-intensiveness
–
–
–
–
Requirements not pre-specifiable
Budgets and schedules not pre-specifiable
Need for evolutionary growth
Need to manage uncertainty and risk
May 19, 2009
Copyright © USC-CSSE
4
University of Southern California
Center for Systems and Software Engineering
Example: SoSE Synchronization Points
FCR1
SoS-Level Exploration
Valuation
Candidate Supplier/
Strategic
Partner n
●
DCR1
OCR1
Architecting
Develop
OCR2

Operation
LCO-type
Proposal &
Feasibility
Info
●
Source
Selection
Rebaseline/
Adjustment FCR1
●
Candidate Supplier/
Strategic Partner 1
OCRx2
OCRx1
System x

Develop
OCRx3
Operation
Operation
OCRx5
OCRx4
Operation
Operation
DCRC
OCRC1

●
●
●
System C   
FCRC
Exploration
Valuation
Architecting
FCRB
System B   
Exploration
Valuation
DCRB
Architecting
FCRA
System A
May 19, 2009
   Exploration
Valuation
Develop
Operation
OCRB1
Develop
Copyright © USC-CSSE

OCRB2
Operation

OCRA1
DCRA
Architecting
OCRC2
Develop
Operation
5

University of Southern California
Center for Systems and Software Engineering
The Broadening Early Cone of
Uncertainty (CU)
• Need greater
investments in
narrowing CU
Global Interactive,
Brownfield
Batch, Greenfield
ConOps
Local Interactive,
Some Legacy
Specs/Plans
IOC
– Mission, investment,
legacy analysis
– Competitive prototyping
– Concurrent engineering
– Associated estimation
methods and
management metrics
• Larger systems will
often have subsystems
with narrower CU’s
May 19, 2009
Copyright © USC-CSSE
6
University of Southern California
Center for Systems and Software Engineering
Current System Acquisition Methods
Easy to misinterpret as one-size-fits-all
• V-Model1
• Spiral Model2
High level guidance assumes that acquirers have extensive acquisition experience...
Without experience, too easy to misinterpret and auger in with disastrous results...
1
2
http://en.wikipedia.org/wiki/V-Model
May 19, 2009
Copyright © USC-CSSE
http://en.wikipedia.org/wiki/Spiral_model
7
University of Southern California
Center for Systems and Software Engineering
Future Process Challenges-II
• Rapid pace of change
– In competition, mission priorities, technology,
Commercial Off-the-Shelf (COTS), environment
– Need incremental development to avoid obsolescence
– Need concurrent vs. sequential processes
– Need both prescience and rapid adaptability
• Software important; humans more important
• Brownfield vs. Greenfield development
– Need to provide legacy continuity of service
– Need to accommodate legacy, OTS constraints
• Always-on, never-fail systems
– Need well-controlled, high-assurance processes
– Need to synchronize and stabilize concurrency
– Need to balance assurance and agility
May 19, 2009
Copyright © USC-CSSE
8
University of Southern California
Center for Systems and Software Engineering
Rapid Change Creates a Late Cone of Uncertainty
– Need incremental vs. one-shot development
4x
Uncertainties in competition,
technology, organizations,
mission priorities
2x
1.5x
1.25x
Relative
Cost Range
x
0.8x
0.67x
0.5x
0.25x
Concept of
Operation
Feasibility
Plans
and
Rqts.
Detail
Design
Spec.
Product
Design
Spec.
Rqts.
Spec.
Product
Design
Detail
Design
Accepted
Software
Devel. and
Test
Phases and Milestones
May
15
July
19,2008
2009
Copyright
©USC-CSSE
© USC-CSSE
99
University of Southern California
Center for Systems and Software Engineering
Tutorial Outline
• Challenges for developing next-generation
software systems
• Overview of ICM
•
•
•
•
Risk-based balance of agility and assurance
ICM process decision table
Guidance and examples for using the ICM
ICM tailoring exercise
May 19, 2009
Copyright © USC-CSSE
10
University of Southern California
Center for Systems and Software Engineering
What is the ICM?
• Risk-driven framework for determining and
evolving best-fit system life-cycle process
• Integrates the strengths of phased and riskdriven spiral process models
• Synthesizes together principles critical to
successful system development
– Commitment and accountability of system sponsors
– Success-critical stakeholder satisficing
– Incremental growth of system definition and
stakeholder commitment
– Concurrent engineering
– Iterative development cycles
– Risk-based activity levels and anchor point milestones
Principles
trump
diagrams…
Principles used by 60-80% of CrossTalk Top-5 projects, 2002-2005
May 19, 2009
Copyright © USC-CSSE
11
University of Southern California
Center for Systems and Software Engineering
ICM Nature and Origins
• Integrates hardware, software, and human factors
elements of systems engineering
– Concurrent exploration of needs and opportunities
– Concurrent engineering of hardware, software, human aspects
– Concurrency stabilized via anchor point milestones
• Developed in response to systems/software issues
– Clarify “spiral development” usage in DoD Instruction 5000.2
• Initial phased version (2005)
– Explain Future Combat System of systems spiral usage to GAO
• Underlying process principles (2006)
– Provide framework for human-systems integration
• National Research Council report (2007)
• Integrates strengths of current process models
– But not their weaknesses
May 19, 2009
Copyright
©USC-CSSE
© USC-CSSE
12
University of Southern California
Center for Systems and Software Engineering
Incremental Commitment in
Gambling
• Total Commitment: Roulette
– Put your chips on a number
• E.g., a value of a key performance parameter
– Wait and see if you win or lose
• Incremental Commitment: Poker, Blackjack
– Put some chips in
– See your cards, some of others’ cards
– Decide whether, how much to commit to
proceed
May
19, 2009
12/31/2007
Copyright
©USC-CSSE
© USC-CSSE
13
13
University of Southern California
Center for Systems and Software Engineering
Scalable Remotely Controlled
Operations
May
19, 2009
12/31/2007
Copyright
©USC-CSSE
© USC-CSSE
14
14
University of Southern California
Center for Systems and Software Engineering
Total vs. Incremental Commitment – 4:1 RPV
• Total Commitment
–
–
–
–
–
Agent technology demo and PR: Can do 4:1 for $1B
Winning bidder: $800M; PDR in 120 days; 4:1 capability in 40 months
PDR: many outstanding risks, undefined interfaces
$800M, 40 months: “halfway” through integration and test
1:1 IOC after $3B, 80 months
• Incremental Commitment [with a number of competing
teams]
–
–
–
–
–
$25M, 6 mo. to VCR [4]: may beat 1:2 with agent technology, but not 4:1
$75M, 8 mo. to FCR [3]: agent technology may do 1:1; some risks
$225M, 10 mo. to DCR [2]: validated architecture, high-risk elements
$675M, 18 mo. to IOC [1]: viable 1:1 capability
1:1 IOC after $1B, 42 months
May
19, 2009
12/31/2007
Copyright
©USC-CSSE
© USC-CSSE
15
15
University of Southern California
Center for Systems and Software Engineering
The Incremental Commitment Life Cycle Process: Overview
Anchor Point
Milestones
Synchronize, stabilize concurrency via FEDs
Risk patterns
determine life
cycle process
May 19, 2009
Copyright © USC-CSSE
16
University of Southern California
Center for Systems and Software Engineering
ICM Activity
Levels for
Complex
Systems
May 19, 2009
Copyright © USC-CSSE
17
University of Southern California
Center for Systems and Software Engineering
Anchor Point Feasibility Evidence Descriptions
• Evidence provided by developer and validated by
independent experts that:
If the system is built to the specified architecture, it will
– Satisfy the requirements: capability, interfaces, level of service, and
evolution
– Support the operational concept
– Be buildable within the budgets and schedules in the plan
– Generate a viable return on investment
– Generate satisfactory outcomes for all of the success-critical
stakeholders
• All major risks resolved or covered by risk management
plans
• Serves as basis for stakeholders’ commitment to proceed
Can be used to strengthen current schedule- or event-based reviews
May 19, 2009
Copyright
©USC-CSSE
© USC-CSSE
18
University of Southern California
Center for Systems and Software Engineering
The Incremental Commitment Life Cycle Process: Overview
Anchor Point
Milestones
Concurrently engr.
OpCon, rqts, arch,
plans, prototypes
May 19, 2009
Copyright © USC-CSSE
Concurrently engr.
Incr.N (ops), N+1
(devel), N+2 (arch)
19
University of Southern California
Center for Systems and Software Engineering
Risk-Driven Scalable Spiral Model:
Increment View
Rapid
Change
Foreseeable
Change
(Plan)
Increment N Baseline
High
Assurance
May 19, 2009
Short Development
Increments
Short, Stabilized
Development
Of Increment N
Increment N Transition/O&M
Stable Development
Increments
Copyright © USC-CSSE
20
University of Southern California
Center for Systems and Software Engineering
Risk-Driven Scalable Spiral Model: Increment View
Unforeseeable Change (Adapt)
Future Increment Baselines
Agile
Rebaselining for
Future Increments
Rapid
Change
Foreseeable
Change
(Plan)
Short
Development
Increments
Increment N Baseline
Stable Development
Increments
High
Assurance
Current V&V
Resources
Continuous V&V
May 19, 2009
Deferrals
Short, Stabilized
Development
of Increment N
Artifacts
Operations and Maintenance
Concerns
Verification and
Validation (V&V)
of Increment N
Copyright © USC-CSSE
Increment N Transition/
Future V&V
Resources
21
University of Southern California
Center for Systems and Software Engineering
Incremental Commitment In Systems and Life:
Anchor Point Milestones
• Common System/Software stakeholder commitment
points
– Defined in concert with Government, industry
organizations
– Initially coordinated with Rational’s Unified Software
Development Process
• Exploration Commitment Review (ECR)
– Stakeholders’ commitment to support initial system
scoping
– Like dating
• Validation Commitment Review (VCR)
– Stakeholders’ commitment to support system concept
definition and investment analysis
– Like going steady
May 19, 2009
Copyright © USC-CSSE
22
University of Southern California
Center for Systems and Software Engineering
Incremental Commitment In Systems and Life:
Anchor Point Milestones (continued)
• Foundations Commitment Review (FCR)
– Stakeholders’ commitment to support system architecting
– Like getting engaged
• Development Commitment Review (DCR)
– Stakeholders’ commitment to support system
development
– Like getting married
• Incremental Operational Capabilities (OCs)
– Stakeholders’ commitment to support operations
– Like having children
May 19, 2009
Copyright © USC-CSSE
23
University of Southern California
Center for Systems and Software Engineering
Example ICM Commercial Application:
Symbiq Medical Infusion Pump
Winner of 2006 HFES Best New Design Award
Described in NRC HSI Report, Chapter 5
May 19, 2009
Copyright © USC-CSSE
24
University of Southern California
Center for Systems and Software Engineering
Symbiq IV Pump ICM Process - I
• Exploration Phase
–
–
–
–
Stakeholder needs interviews, field observations
Initial user interface prototypes
Competitive analysis, system scoping
Commitment to proceed
• Valuation Phase
–
–
–
–
–
Feature analysis and prioritization
Display vendor option prototyping and analysis
Top-level life cycle plan, business case analysis
Safety and business risk assessment
Commitment to proceed while addressing risks
May 19, 2009
Copyright © USC-CSSE
25
University of Southern California
Center for Systems and Software Engineering
Symbiq IV Pump ICM Process - II
• Architecting Phase
–
–
–
–
–
–
Modularity of pumping channels
Safety feature and alarms prototyping and iteration
Programmable therapy types, touchscreen analysis
Failure modes and effects analyses (FMEAs)
Prototype usage in teaching hospital
Commitment to proceed into development
• Development Phase
–
–
–
–
Extensive usability criteria and testing
Iterated FMEAs and safety analyses
Patient-simulator testing; adaptation to concerns
Commitment to production and business plans
May 19, 2009
Copyright © USC-CSSE
26
University of Southern California
Center for Systems and Software Engineering
May 19, 2009
Copyright © USC-CSSE
27
University of Southern California
Center for Systems and Software Engineering
May 19, 2009
Copyright © USC-CSSE
28
University of Southern California
Center for Systems and Software Engineering
May 19, 2009
Copyright © USC-CSSE
29
University of Southern California
Center for Systems and Software Engineering
ICM Summary
• Current processes not well matched to future challenges
– Emergent, rapidly changing requirements
– High assurance of scalable performance and qualities
• Incremental Commitment Model addresses challenges
– Assurance via evidence-based milestone commitment reviews,
stabilized incremental builds with concurrent V&V
• Evidence shortfalls treated as risks
– Adaptability via concurrent agile team handling change traffic and
providing evidence-based rebaselining of next-increment specifications
and plans
– Use of critical success factor principles: stakeholder satisficing,
incremental growth, concurrent engineering, iterative development, riskbased activities and milestones
• Major implications for funding, contracting, career paths
May 19, 2009
Copyright © USC-CSSE
30
University of Southern California
Center for Systems and Software Engineering
Tutorial Outline
• Challenges for developing next-generation
software systems
• Overview of ICM
• Risk-based balance of agility and
assurance
• ICM process decision table
• Guidance and examples for using the ICM
• ICM tailoring exercise
May 19, 2009
Copyright © USC-CSSE
31
University of Southern California
Center for Systems and Software Engineering
Using Risk to Balance
Assurance and Agility - Overview
Step 1.
Risk Analysis
Plan-driven risks
dominate
Step 2.
Risk
Comparison
Rate the project’s
environmental, agilityoriented and plan-driven
risks.
Compare
the agile
and Plandriven risks
Agility risks
dominate
Go Risk-based
Agile
Go Risk-based
Plan-driven
Neither dominate
Uncertain
about
ratings?
Yes
Buy information via
prototyping, data
collection and analysis
No
Step 3.
Architecture
Analysis
Architect application to
encapsulate agile parts
Go Risk-based
Agile in agile
parts; Go Riskbased Plandriven elsewhere
Step 5.
Execute and Monitor
Note: Feedback
loops present,
but omitted for
simplicity
May 19, 2009
Deliver incremental
capabilities according to
strategy
Monitor progress and
risks/opportunities,
readjust balance and
process as appropriate
Copyright © USC-CSSE
Tailor life cycle process
around risk patterns
and anchor point
commitment milestones
Step 4.
Tailor Life Cycle
32
University of Southern California
Center for Systems and Software Engineering
Risks Used in Risk Comparison
• Environmental risks
–
–
–
–
Technology uncertainties
Many stakeholders
Complex system-of-systems
Legacy compatibility
• Agility risks
–
–
–
–
–
Scalability
Use of simple design
Personnel turnover
Too-frequent releases
Not enough agile-capable people
• Plan-driven risks
–
–
–
–
Rapid change
Need for rapid results
Emergent requirements
Not enough discipline-capable people
May 19, 2009
Copyright © USC-CSSE
33
University of Southern California
Center for Systems and Software Engineering
Two Examples
• Two agent-based systems
– Small – Event Managers application
– Large – National Information System for Crisis
Management (NISCM) application
• Each example results in a development
strategy based on risk analyses
May 19, 2009
Copyright © USC-CSSE
34
University of Southern California
Center for Systems and Software Engineering
Event Managers Project
• Small startup company
• Diverse set of smaller agent-based
planning applications
• This project: support for managing the
infrastructure and operations of
conferences and conventions
• Widening variety of options and
interdependencies, makes an agent-based
approach highly attractive
May 19, 2009
Copyright © USC-CSSE
35
University of Southern California
Center for Systems and Software Engineering
Event Managers Home Ground Chart
May 19, 2009
Copyright © USC-CSSE
36
University of Southern California
Center for Systems and Software Engineering
Event Managers Risks
May 19, 2009
Copyright © USC-CSSE
37
University of Southern California
Center for Systems and Software Engineering
Event Managers Mitigations
May 19, 2009
Copyright © USC-CSSE
38
University of Southern California
Center for Systems and Software Engineering
Event Managers Strategy
1. Startup
Teambuilding and Shared
Vision
2. Design, Development and Deployment
Customer
Management;
Representative
Users
Joint DeveloperCustomer Agile Team
• Establish CRACK joint
management team
• Develop shared vision,
results chain, life cycle
strategy, infrastructure
choices
• Establish commitment
to proceed, incremental
capabilities, backlog
• Test, exercise, deploy new increment
• Re-prioritize next-increment features
• Analyze lessons learned; prepare
strategy for next increment
• Develop next-increment
features
LCA
May 19, 2009
Copyright © USC-CSSE
39
University of Southern California
Center for Systems and Software Engineering
NISCM Profile
• Broad coalition of government agencies
and critical private-sector organizations
• Support cross-agency and public-private
sector coordination of crisis management
activities
• Adaptive mobile network, virtual
collaboration, information assurance, and
information integration technology
• Private-sector system-of-systems
integration contractor
May 19, 2009
Copyright © USC-CSSE
40
University of Southern California
Center for Systems and Software Engineering
NISCM Home Ground Chart
May 19, 2009
Copyright © USC-CSSE
41
University of Southern California
Center for Systems and Software Engineering
NISCM Risks
May 19, 2009
Copyright © USC-CSSE
42
University of Southern California
Center for Systems and Software Engineering
NISCM Mitigations
May 19, 2009
Copyright © USC-CSSE
43
University of Southern California
Center for Systems and Software Engineering
NISCM Strategy
Startup
Furnish CRACK
representatives
and alternates
Stakeholders
• Staff and organize
to cover all
success-critical
risk areas
• Prepare for and
select competitors
for concept
definition
Teambuilding
and Shared
Vision
Systems
Definition and
Architecting
• Develop results chains
• Identify missing
stakeholders
• Enhance understanding
• Explore goals, options,
technology, prototypes,
risks
• Negotiate mutually
satisfactory objectives
and priorities
• Formulate top-level
operational concept,
requirements,
architecture, life cycle
plan, feasibility rational
• Select winning system
integration contract
• Prepare for/select
competitors for
component subcontract
definition
• Elaborate operational
concept, prototypes,
transition strategy
• Evaluate/determine
COTS, reuse, and
architecture choices
• Develop/ validate criticalinfrastructure software
• Prioritize/sequence
desired capabilities,
outstanding risks
• Formulate definitive
operational concept,
requirements,
architecture, life cycle
plan, feasibility rationale
• Use to solicit/select
component
subcontractors
Hold Life
Cycle
Objective
Review. If
feasibility
rationale is
valid,
commit to
proceed
NSO and I3-led Project
Risk Management Team
• Ensure representative
exercise of incremental
capabilities
• Monitor progress and new
operational development
Hold Life
Cycle
Architecture
Review. If
feasibility
rationale is
valid,
commit to
proceed
• Develop compatible
component-level
prototypes, requirements,
architectures, plans,
critical software, feasibility
rationale
• Classify modules by likely
stability and criticality
Stable, Higher-criticality
Module Teams
Dynamic, Lower-criticality
Module Teams
May 19, 2009
Design, Development and
Deployment
LCO
• Use risk to determine content of
artifacts
• Monitor project progress,
risk resolution, and new
technology developments
• Identify/work critical projectlevel actions
• Continuously integrate/ test
growing software
infrastructure and
components
Organize
into
disciplined
and agile
module
teams
Copyright © USC-CSSE
LCA
• Communicate
proposed
redirections
• Prepare for
and execute
acceptance
tests,
installation,
operational
evolution
• Analyze
feedback on
current
version
• Reprioritize
features
• Prepare
strategy for
next
increment
Plan, specify,
develop stable,
higher-criticality
module
increments
Plan, specify,
develop dynamic,
lower-criticality
module
increments
44
University of Southern California
Center for Systems and Software Engineering
Risk Profiles Across Agent-Based Examples
May 19, 2009
Copyright © USC-CSSE
45
University of Southern California
Center for Systems and Software Engineering
Tutorial Outline
• Challenges for developing next-generation
software systems
• Overview of ICM
• Risk-based balance of agility and assurance
• ICM process decision table
• Guidance and examples for using the ICM
• ICM tailoring exercise
May 19, 2009
Copyright © USC-CSSE
46
University of Southern California
Center for Systems and Software Engineering
The Incremental Commitment Life Cycle Process: Overview
Anchor Point
Milestones
Synchronize, stabilize concurrency via FEDs
Risk patterns
determine life
cycle process
May 19, 2009
Copyright © USC-CSSE
47
University of Southern California
Center for Systems and Software Engineering
The ICM as Risk-Driven Process
Generator
• Stage I of the ICM has 3 decision nodes with 4 options/node
– Culminating with incremental development in Stage II
– Some options involve go-backs
– Results in many possible process paths
• Can use ICM risk patterns to generate frequently-used
processes
– With confidence that they fit the situation
• Can generally determine this in the Exploration phase
– Develop as proposed plan with risk-based evidence at VCR
milestone
– Adjustable in later phases
May 19, 2009
Copyright © USC-CSSE
48
University of Southern California
Center for Systems and Software Engineering
Different Risk Patterns Yield Different Processes
May
19, 2009
03/19/2008
Copyright
©USC-CSSE
© USC-CSSE
49
49
University of Southern California
Center for Systems and Software Engineering
The ICM Process Decision Table:
Key Decision Inputs
•
•
•
•
Product and project size and complexity
Requirements volatility
Mission criticality
Nature of Non-Developmental Item (NDI)* support
– Commercial, open-source, reused components
• Organizational and Personnel Capability
* NDI Definition [DFARS]: a) any product that is available in the commercial marketplace; b) any
previously developed product in use by a U.S. agency (federal, state, or local) or a foreign government that
has a mutual defense agreement with the U.S.; c) any product described in the first two points above that
requires only modifications to meet requirements; d) any product that is being produced, but not yet in the
commercial marketplace, that satisfies the above criteria.
May 19, 2009
Copyright © USC-CSSE
50
University of Southern California
Center for Systems and Software Engineering
The ICM Process Decision Table:
Key Decision Outputs
• Key Stage I activities: incremental definition
• Key Stage II activities: incremental
development and operations
• Suggested calendar time per build, per
deliverable increment
May 19, 2009
Copyright © USC-CSSE
51
University of Southern California
Center for Systems and Software Engineering
Common Risk-Driven Special Cases of the ICM (Cases 1-4)
Case 1: Use NDI
Case 2: Agile
Example: Small accounting system
Size, Complexity: Size variable, complexity low
Typical Change Rate/Month: Negligible
Criticality: n/a
NDI Support: Complete
Organizational Personnel Capability: NDI-experienced (medium)
Key Stage I Activities (Incremental Definition): Acquire NDI
Key Stage II Activities (Incremental Development/Operations): Use
NDI
Time/Build: n/a
Time/Increment: Vendor-driven
Example: E-services
Size, Complexity: Low
Typical Change Rate/Month: 1-30%
Criticality: Low to medium
NDI Support: Good, in place
Organizational Personnel Capability: Agile-ready, medium-high
experience
Key Stage I Activities (Incremental Definition): Skip Valuation and
Architecting phases
Key Stage II Activities (Incremental Development/Operations): Scrum
plus agile methods of choice
Time/Build: <= 1 day
Time/Increment: 2-6 weeks
Case 3: Architected Agile
Case 4: Formal Methods
Example: Business data processing
Size, Complexity: Medium
Typical Change Rate/Month: 1-10 %
Criticality: Medium to high
NDI Support: Good, most in place
Organizational Personnel Capability: Agile-ready, medium to high
experience
Key Stage I Activities (Incremental Definition): Combine Valuation,
Architecting phases. Complete NDI preparation.
Key Stage II Activities (Incremental Development/Operations):
Architecture-based Scrum of Scrums
Time/Build: 2-4 weeks
Time/Increment: 2-6 months
May 19, 2009
Example: Security kernel; Safety-critical LSI chip
Size, Complexity: Low
Typical Change Rate/Month: 0.3%
Criticality: Extra high
NDI Support: None
Organizational Personnel Capability: Strong formal methods experience
Key Stage I Activities (Incremental Definition): Precise formal
specification
Key Stage II Activities (Incremental Development/Operations):
Formally-based programming language; formal verification
Time/Build: 1-5 days
Time/Increment: 1-4 weeks
Copyright © USC-CSSE
52
University of Southern California
Center for Systems and Software Engineering
Common Risk-Driven Special Cases of the ICM (Cases 5-8)
Case 5: Hardware with Embedded Software Component
Example: Multi-sensor control device
Size, Complexity: Low
Typical Change Rate/Month: 0.3 - 1 %
Criticality: Medium to very high
NDI Support: Good, in place
Organizational Personnel Capability: Experienced, medium-high
Key Stage I Activities (Incremental Definition): Concurrent
hardware/software engineering. CDR-level ICM DCR
Key Stage II Activities (Incremental Development/Operations): IOC
development, LRIP, FRP. Concurrent version N+1 engineering
Time/Build: Software 1-5 days
Time/Increment: Market-driven
Case 7: NDI-Intensive
Case 8: Hybrid Agile/Plan-Driven System
Example: Supply chain management
Size, Complexity: Medium to high
Typical Change Rate/Month: 0.3 – 3%
Criticality: Medium to very high
NDI Support: NDI-driven architecture
Organizational Personnel Capability: NDI-experienced, medium to
high
Key Stage I Activities (Incremental Definition): Thorough NDI-suite
life cycle cost-benefit analysis, selection, concurrent
requirements/architecture definition
Key Stage II Activities (Incremental Development/Operations): Proactive NDI evolution influencing, NDI upgrade synchronization
Time/Build: Software: 1-4 weeks
Time/Increment: Systems: 6-18 months
May 19, 2009
Case 6: Indivisible IOC
Example: Complete vehicle platform
Size, Complexity: Medium to high
Typical Change Rate/Month: 0.3 – 1%
Criticality: High to very high
NDI Support: Some in place
Organizational Personnel Capability: Experienced, medium to high
Key Stage I Activities (Incremental Definition): Determine minimumIOC likely, conservative cost. Add deferrable software features as
risk reserve
Key Stage II Activities (Incremental Development/Operations): Drop
deferrable features to meet conservative cost. Strong award free for
features not dropped.
Time/Build: Software: 2-6 weeks
Time/Increment: Platform: 6-18 months
Example: C4ISR system
Size, Complexity: Medium to very high
Typical Change Rate/Month: Mixed parts; 1-10%
Criticality: Mixed parts; Medium to very high
NDI Support: Mixed parts
Organizational Personnel Capability: Mixed parts
Key Stage I Activities (Incremental Definition): Full ICM, encapsulated
agile in high change, low-medium criticality parts (Often HMI,
external interfaces)
Key Stage II Activities (Incremental Development/Operations): Full
ICM, three-team incremental development, concurrent V&V, nextincrement rebaselining
Time/Build: 1-2 months
Time/Increment: 9-18 months
Copyright © USC-CSSE
53
University of Southern California
Center for Systems and Software Engineering
Common Risk-Driven Special Cases of the ICM (Cases 9-11)
Case 9: Multi-Owner Directed System of Systems
Example: Net-centric military operations
Size, Complexity: Very high
Typical Change Rate/Month: Mixed parts; 1-10 %
Criticality: Very high
NDI Support: Many NDIs, some in place
Organizational Personnel Capability: Related experience, medium to
high
Key Stage I Activities (Incremental Definition): Full ICM; extensive
multi-owner team building, negotiation
Key Stage II Activities (Incremental Development/Operations):
Full ICM; large ongoing system/software engineering effort
Time/Build: 2-4 months
Time/Increment: 18-24 months
Case 10: Family of Systems
Example: Medical device product line
Size, Complexity: Medium to very high
Typical Change Rate/Month: 1-3%
Criticality: Medium to very high
NDI Support: Some in place
Organizational Personnel Capability: Related experience, medium to
high
Key Stage I Activities (Incremental Definition): Skip Valuation and
Architecting phases
Key Stage II Activities (Incremental Development/Operations):
Scrum plus agile methods of choice
Time/Build: 1-2 months
Time/Increment: 9-18 months
Case 11: Brownfield
Example: Incremental legacy phaseout
Size, Complexity: High to very high
Typical Change Rate/Month: 0.3-3%
Criticality: Medium-high
NDI Support: NDI as legacy replacement
Organizational Personnel Capability: Legacy re-engineering
Key Stage I Activities (Incremental Definition): Re-engineer/refactor legacy into services
Key Stage II Activities (Incremental Development/Operations): Incremental legacy phaseout
Time/Build: 2-6 weeks/refactor
Time/Increment: 2-6 months
May 19, 2009
Copyright © USC-CSSE
54
University of Southern California
Center for Systems and Software Engineering
Common Risk-Driven Special Cases of the ICM (Cases 12a/b)
Case 12a: Net-Centric Services – Community
Support
Case 12b: Net-Centric Services – Quick Response
Decision Support
Example: Community services or special interest group
Size, Complexity: Low to medium
Typical Change Rate/Month: 0.3-3%
Criticality: Low to medium
NDI Support: Tailorable service elements
Organizational Personnel Capability: NDI-experienced
Key Stage I Activities (Incremental Definition): Filter, select,
compose, tailor NDI
Key Stage II Activities (Incremental Development/Operations):
Evolve tailoring to meet community needs
Time/Build: <= 1 day
Time/Increment: 2-12 months
Example: Response to competitor initiative
Size, Complexity: Medium to high
Typical Change Rate/Month: 3-30%
Criticality: Medium to high
NDI Support: Tailorable service elements
Organizational Personnel Capability: NDI-experienced
Key Stage I Activities (Incremental Definition): Filter, select,
compose, tailor NDI
Key Stage II Activities (Incremental Development/Operations):
Satisfy quick response; evolve or phase out
Time/Build: <= 1 day
Time/Increment: Quick response-driven
LEGEND
C4ISR: Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance.
CDR: Critical Design Review.
DCR: Development Commitment Review.
FRP: Full-Rate Production.
HMI: Human-Machine Interface.
HW: Hard ware.
IOC: Initial Operational Capability.
LSI: Large Scale Integration.
LRIP: Low-Rate Initial Production.
NDI: Non-Development Item.
SW: Software
May 19, 2009
Copyright © USC-CSSE
55
University of Southern California
Center for Systems and Software Engineering
Tutorial Outline
•
•
•
•
Motivation
Overview of ICM
Risk-based balance of agility and assurance
ICM process decision table
• Guidance and examples for using
the ICM process decision table
• ICM tailoring exercise
May 19, 2009
Copyright © USC-CSSE
56
University of Southern California
Center for Systems and Software Engineering
Case 3: Architected Agile
•
Exploration phase determines
– Need to accommodate fairly rapid change, emergent requirements, early user
capability
– Low risk of scalability up to 100 people
– NDI support of growth envelope
– Nucleus of highly agile-capable personnel
– Moderate to high loss due to increment defects
•
•
•
•
•
•
•
•
•
Example: Business data processing
Size/complexity: Medium
Anticipated change rate (% per month): 1-10%
Criticality: Medium to high
NDI support: Good, most in place
Organizational and personnel capability: Agile-ready, med-high capability
Key Stage I activities: Combined Valuation and Architecting phase,
complete NDI preparation
Key Stage II activities: Architecture-based scrum of scrums
Time/build: 2-4 weeks
Time/increment: 2-6 months
May 19, 2009
Copyright © USC-CSSE
57
University of Southern California
Center for Systems and Software Engineering
USA Medical Case Study
• 1400 software people; 7M SLOC; 7 sites
– 4 in Europe, 2 in India
• 500 medical applications; 500 financial; others
• Survivability-critical software problems
– Reliability, productivity, performance, interoperability
– Sarbanes-Oxley requirements
– Management receptive to radical change
• Some limited experimental use of agile methods
– Led by top software technologist/manager
• Committed to total change around Scrum and XP
May 19, 2009
Copyright © USC-CSSE
58
University of Southern California
Center for Systems and Software Engineering
USA Medical Adoption Profile
• July 2004 - July 2005
100
90
80
70
60
50
40
30
20
10
0
Scrum
Teams
– Recruit top people from all
sites into core team(s)
– Get external expert help
– Develop architecture
– Early Scrum successes with
infrastructure
– Revise policies and practices
– Train, reculture everyone
– Manage expectations
• July 2005 – July 2006
2004 2005 2006 2007
Jul Jul Jul Mar
May 19, 2009
– Begin full-scale development
– Core teams as mentors
Copyright © USC-CSSE
59
University of Southern California
Center for Systems and Software Engineering
Architected Agile – USA Medical
• Include customers and marketers
– New roles; do’s/don’ts/opportunities; CRACK personnel; full
collaboration and teamwork; expectations management
• Scrum; most XP practices; added company practices
– 6-12 person teams with team rooms, dedicated servers
– Hourly smoke test; nightly build and regression test
– Just-in-time analysis; story-point estimates; fail fast; detailed short-term
plans; company architecture compliance
– Embrace change in applications and practices
– Global teams: wikis, daily virtual meetings, act as if next-door
• Release management
– 2-12 week architecting Sprint Zero; 3-10 1-month Sprints; Release Sprint;
1-6 month beta test
– Next Sprint Zero concurrent with Release Sprint
• Initiative manager and team
– Define practices; evolve infrastructure; provide training; guide
implementation; evaluate compliance/usage; continuous improvement
May 19, 2009
Copyright © USC-CSSE
60
University of Southern California
Center for Systems and Software Engineering
Case 7: NDI-Intensive
•
•
•
•
•
•
•
•
•
•
•
Biggest risks: incompatible NDI; rapid change, business/mission
criticality; low NDI assessment and integration experience; supply chain
stakeholder incompatibilities
Example: Supply chain management
Size/complexity: Medium to high
Anticipated change rate (% per month): 0.3-3%
Criticality: Medium to very high
NDI support: NDI-driven architecture
Organizational and personnel capability: NDI-experienced; medium to
high capability
Key Stage I activities: Thorough NDI-suite life cycle cost-benefit
analysis, selection, concurrent requirements and architecture definition
Key Stage II activities: Pro-active NDI evolution influencing, NDI upgrade
synchronization
Time/build: 1-4 weeks (software)
Time/increment: 6-18 months (systems)
May 19, 2009
Copyright © USC-CSSE
61
University of Southern California
Center for Systems and Software Engineering
COTS-NDI Decision Framework
Start
P7: Custom Development
P1: Identify OC & P's: Evaluation
Criteria, Weights and Scenarios
C
No
Yes
Deploy
P6: Can Adjust
OC & P's?
P2: Identify alternatives:
Candidate COTS products
None Acceptable
A
P3: Assess Candidate COTS products
Full COTS solution best
No Application Code
or Glue Code Required
P4: Tailoring
Required?
Yes
Partial COTS
Solution Best
P8: Coordinate
Application Code
development and
Glue Code effort
P5: Multiple COTS
cover all OC & P’s?
No
Yes
C
No
G
P9: Develop
Custom Code
P10: Develop
Glue Code
T
P11: Tailor
COTS
products
May 19, 2009
P12: Productize,
Transition
Application
Deploy
Copyright © USC-CSSE
62
University of Southern California
Center for Systems and Software Engineering
Assessment Process Elements
A1: Initial f iltering of
candidates with
respect to criteria
P6: Can
Adjust the OC
& P’s
None
Acceptable
G
Candidates remaining
None
Acceptable
P5: Multiple COTS
cov er OC & P’s?
Partial-COTS
solution best
Yes, Glue Code
A2: Need tailoring or
glue code f or
ev aluation
A3: Ev aluate
remaining COTS
candidates
No
Yes,
Tailor
T
May 19, 2009
Full-COTS
solution
best
P4: Tailoring
Required?
Tailoring or Glue
Code uncertainties
Copyright © USC-CSSE
63
University of Southern California
Center for Systems and Software Engineering
Example CBA: Oversize Image Viewer
• Client:
– A librarian at USC
• Developers:
– A development team of 5 graduates at USC
• Main tasks
– Development of a viewing capability for
oversized images, including features such as
• Image navigation, cataloging, search, archive, and
access administration
• Application of CBA decision framework
– First three ICM phases
– Sequencing of tasks driven by stakeholders’
win conditions and projects major risk items
May 19, 2009
Copyright © USC-CSSE
64
University of Southern California
Center for Systems and Software Engineering
ICM Application to Oversize Image Viewer
Stakeholders
OC&P’s
Alternatives
Evaluation;
Risks
Risk Addressed
Risk Resolution
Product
Elaboration
Process
Elaboration
Stakeholder
Review
Commitment
Exploration
Developer, customer, library
-user client,
COTS vendors
Image navigation, cataloguing, search,
archive and access ad
ministration
COTS cost  $25K,  5 user
organizations
IOC developed, transitioned in 24 weeks
ER Mapper, Mr SID, Systems ABC, XYZ
XYZ > $25K; ABC < 5 user org’s
ER Mapper, Mr SID acceptable
Risk picking wrong product without
exercise
Exercise ER Mapper, Mr SID
ER Mapper image navigation, display
stronger
Use ER Mapper for image navigation,
display
Tailor ER Mapper for library-user
Windows client
Customer: want campus
-wide usage,
support of Unix, Mac platforms
ER Mapper runs only on Windows
Customer will find Unix, Mac user
community representatives
May 19, 2009
Valuation
Foundations
Additional user representatives (Unix, Mac Additional end-users (staff, students) for
communities)
usability evaluation
System usable on Windows, Unix, and
Detailed GUI’s satisfy representative users
Mac platforms
ER Mapper, Mr SID
ER Mapper Windows-only; plans to
support Unix, Mac; schedule unclear
Mr SID supports all 3 platforms
Risk of Unix, Mac non
- support
Ask ER Mapper for guaranteed Unix, Mac
support in 9 months
ER Mapper: no guaranteed Unix, Mac
support even in 18 months
Use Mr SID for image navigation, MySQL
for catalog support, Java for admin/GUI
support
Prepare totailor Mr SID, My SQL to
support all 3 platforms
Need to address Mr SID/My SQL/Java
interoperability, glue code issues; GUI
usability issues
Many GUI alternatives
Risk of developing wrong GUI without end
-user
prototyping
Mr SID/MY SQL/Java interoperability risks
Customer will buy Mr SID
Customer will commit to post
-deployment
support of software
Users will commit to support training,
installation, operations
Users will support GUI prototype
evaluations
Copyright © USC-CSSE
Prototype full range of system GUI’s, Mr SID/My
SQL/Java interfaces
Acceptable GUI’s, Mr SID/My SQL/.Java
interfaces determined
Develop production Mr SID/My SQL/Java glue
code
Use Schedule as Independent Variable (SAIV)
process to ensure acceptable IOC in 24 weeks
Need to prioritize desired capabilities to support
SAIV process
65
University of Southern California
Center for Systems and Software Engineering
Use of the CBA Process Framework: Exploration
Start
P7: Custom Development
P1: Identify OC & P's: Evaluation
Criteria, Weights and Scenarios
Assessing
COTS’s
for image
viewing
capabilities
P6: Can Adjust
OC & P's?
P2: Identify alternatives:
Candidate COTS products
None Acceptable
A
P3: Assess Candidate COTS products
Full COTS solution best
No Application Code
or Glue Code Required
P4: Tailoring
Required?
Yes
C
No
Yes
Deploy
New objectives
Partial COTS
Solution Best
P8: Coordinate
Application Code
development and
Glue Code effort
P5: Multiple COTS
cover all OC & P’s?
No
Yes
C
No
G
P9: Develop
Custom Code
P10: Develop
Glue Code
T
P11: Tailor
COTS
products
May 19, 2009
P12: Productize,
Transition
Application
Deploy
Copyright © USC-CSSE
66
University of Southern California
Center for Systems and Software Engineering
Composing the Assessment Element
in Exploration Phase
A1: Initial f iltering of
candidates with
respect to criteria
P6: Can
Adjust the OC
& P’s
None
Acceptable
G
Candidates remaining
None
Acceptable
P5: Multiple COTS
cov er OC & P’s?
Partial-COTS
solution best
Y es, Glue Code
A2: Need tailoring or
glue code f or
ev aluation
A3: Ev aluate
remaining COTS
candidates
No
Y es,
Tailor
T
May 19, 2009
Full-COTS
solution
best
P4: Tailoring
Required?
Tailoring or Glue
Code uncertainties
Copyright © USC-CSSE
67
University of Southern California
Center for Systems and Software Engineering
Use of the CBA Process Framework: Valuation
Start
P7: Custom Development
P1: Identify OC & P's: Evaluation
Criteria, Weights and Scenarios
Assessing
COTS’s
for other
features
C
No
Yes
P6: Can Adjust
OC & P's?
P2: Identify alternatives:
Candidate COTS products
None Acceptable
A
P3: Assess Candidate COTS products
Full COTS solution best
No Application Code
or Glue Code Required
P4: Tailoring
Required?
Yes
Partial COTS
Solution Best
P8: Coordinate
Application Code
development and
Glue Code effort
Deploy
Assessing COTS’s
for additional objectives
P5: Multiple COTS
cover all OC & P’s?
No
Yes
C
No
G
P9: Develop
Custom Code
P10: Develop
Glue Code
T
P11: Tailor
COTS
products
May 19, 2009
P12: Productize,
Transition
Deploy
Application
Copyright © USC-CSSE
68
University of Southern California
Center for Systems and Software Engineering
Use of the CBA Process Framework: Foundations
Start
P7: Custom Development
P1: Identify OC & P's: Evaluation
Criteria, Weights and Scenarios
C
No
Yes
Deploy
P6: Can Adjust
OC & P's?
P2: Identify alternatives:
Candidate COTS products
None Acceptable
A
P3: Assess Candidate COTS products
Full COTS solution best
No Application Code
or Glue Code Required
P4: Tailoring
Required?
Yes
Partial COTS
Solution Best
P8: Coordinate
Application Code
development and
Glue Code effort
P5: Multiple COTS
cover all OC & P’s?
No
Yes
C
No
G
P9: Develop
Custom Code
P10: Develop
Glue Code
Detailed
assessment;
tailoring and
glue coding to
help assessing
interoperability
T
P11: Tailor
COTS
products
May 19, 2009
P12: Productize,
Transition
Application
Deploy
Copyright © USC-CSSE
69
University of Southern California
Center for Systems and Software Engineering
Case 11: Brownfield
•
•
•
•
•
•
•
•
•
•
Example: Incremental legacy phaseout
Size/complexity: High to very high
Anticipated change rate (% per month): 0.3-3
Criticality: Medium-high
NDI support: NDI as legacy replacement
Organizational and personnel capability: Legacy re-engineering
Key Stage I activities: Re-engineer/refactor legacy into services
Key Stage II activities: Incremental legacy phaseout
Time/build: 2-6 week/refactor
Time/increment: 2-6 months
May 19, 2009
Copyright © USC-CSSE
70
University of Southern California
Center for Systems and Software Engineering
ICM and Brownfield Development
• Many process models are Greenfieldoriented
– Requirements→Design→Develop→Test→Operate
• Failed Greenfield project example
– Corporate central financial system
– To replace spaghetti-code collection of COBOL
programs
• Improved ICM Brownfield approach
– Concurrent new-system definition and legacy
system re-engineering
May 19, 2009
Copyright © USC-CSSE
71
University of Southern California
Center for Systems and Software Engineering
Failed Greenfield Corporate
Financial System
• Used waterfall approach
– Gathered requirements
– Chose best-fit ERP system
– Provided remaining enhancements
• Needed to ensure continuity of service
– Planned incremental phase-in of new services
• Failed due to inability to selectively phase
out legacy services
– Dropped after 2 failed tries at cost of $40M
May 19, 2009
Copyright © USC-CSSE
72
University of Southern California
Center for Systems and Software Engineering
Legacy Systems Patched, Highly Coupled
Financial and Non-Financial Services
Legacy Business Services
Project Services
Contract Services
Deliverables
Management Billing
Staffing
Budgeting
Work Breakdown Structure
Earned Value Management
Subcontracting
Scheduling
Progress Tracking
Change Tracking
Reqs, Configuration Management
May 19, 2009
Copyright © USC-CSSE
73
University of Southern California
Center for Systems and Software Engineering
ICM Approach to Brownfield Engineering
• Understanding needs
– Analysis of legacy system difficulties
• Envisioning opportunities
– Concurrently decouple legacy financial and non-financial
services, explore new system phase-in and architecture
options
• System scoping and architecting
– Extract legacy financial, non-financial services
– Prioritize, plan for incremental financial services phase-in/out
• Feasibility evidence development
– Successful examples of representative service extractions
– Evidence of cost, schedule, performance feasibility
May 19, 2009
Copyright © USC-CSSE
74
University of Southern California
Center for Systems and Software Engineering
Result of Legacy Re-engineering
Legacy Business Services
Contract Services
Contract
Financial
Services
•Billing
•Subcontract
payments
•...
May 19, 2009
Contract NonFinancial
Services
•Deliverables
mgmt.
•Terms
compliance
•...
Project Services
General
Financial
Services
•Accounting
•Budgeting
•Earned
value
•Payroll
•...
General NonFinancial
Services
•Progress
tracking
•Change
tracking
•...
Copyright © USC-CSSE
Project
Financial
Services
•WBS
•Expenditure
categories
•...
Project NonFinancial
Services
•Scheduling
•Staffing
•Reqs CM
•...
75
University of Southern California
Center for Systems and Software Engineering
Frequently Asked Question
• Q: Having all that ICM generality and then using the
decision table to come back to a simple model seems
like an overkill.
– If my risk patterns are stable, can’t I just use the special case
indicated by the decision table?
• A: Yes, you can and should – as long as your risk
patterns stay stable. But as you encounter change,
the ICM helps you adapt to it.
– And it helps you collaborate with other organizations that
may use different special cases.
May 19, 2009
Copyright © USC-CSSE
76
University of Southern California
Center for Systems and Software Engineering
Tutorial Outline
• Challenges for developing next-generation
software systems
• Overview of ICM
• Risk-based balance of agility and assurance
• ICM process decision table
• Guidance and examples for using the ICM
• ICM tailoring exercise
May 19, 2009
Copyright © USC-CSSE
77
University of Southern California
Center for Systems and Software Engineering
ICM Tailoring Exercise:
SupplyChain.com agent-based system
1. Analyze project and determine key decision drivers
2. Based on key decision drivers, identify candidate ICM
special case for development
3. Evaluate environment, agile, and plan-driven risks,
and associated risk mitigation strategies
May 19, 2009
Copyright © USC-CSSE
78
University of Southern California
Center for Systems and Software Engineering
SupplyChain.com Profile
• Turnkey agent-based supply chain
management systems
• Distributed, multi-organization teams of
about 50 mostly-experienced people
• Parts of applications are relatively stable,
while others are highly volatile
• Architectures are driven by a few key COTS
packages that are also continually evolving
May 19, 2009
Copyright © USC-CSSE
79
University of Southern California
Center for Systems and Software Engineering
The ICM Process Decision Table:
Key Decision Inputs
•
•
•
•
Product and project size and complexity
Requirements volatility
Mission criticality
Nature of Non-Developmental Item (NDI)* support
– Commercial, open-source, reused components
• Organizational and Personnel Capability
* NDI Definition [DFARS]: a) any product that is available in the commercial marketplace; b) any
previously developed product in use by a U.S. agency (federal, state, or local) or a foreign government that
has a mutual defense agreement with the U.S.; c) any product described in the first two points above that
requires only modifications to meet requirements; d) any product that is being produced, but not yet in the
commercial marketplace, that satisfies the above criteria.
May 19, 2009
Copyright © USC-CSSE
80
University of Southern California
Center for Systems and Software Engineering
Common Risk-Driven Special Cases of the ICM (Cases 1-4)
Case 1: Use NDI
Case 2: Agile
Example: Small accounting system
Size, Complexity: Size variable, complexity low
Typical Change Rate/Month: Negligible
Criticality: n/a
NDI Support: Complete
Organizational Personnel Capability: NDI-experienced (medium)
Key Stage I Activities (Incremental Definition): Acquire NDI
Key Stage II Activities (Incremental Development/Operations): Use
NDI
Time/Build: n/a
Time/Increment: Vendor-driven
Example: E-services
Size, Complexity: Low
Typical Change Rate/Month: 1-30%
Criticality: Low to medium
NDI Support: Good, in place
Organizational Personnel Capability: Agile-ready, medium-high
experience
Key Stage I Activities (Incremental Definition): Skip Valuation and
Architecting phases
Key Stage II Activities (Incremental Development/Operations): Scrum
plus agile methods of choice
Time/Build: <= 1 day
Time/Increment: 2-6 weeks
Case 3: Architected Agile
Case 4: Formal Methods
Example: Business data processing
Size, Complexity: Medium
Typical Change Rate/Month: 1-10 %
Criticality: Medium to high
NDI Support: Good, most in place
Organizational Personnel Capability: Agile-ready, medium to high
experience
Key Stage I Activities (Incremental Definition): Combine Valuation,
Architecting phases. Complete NDI preparation.
Key Stage II Activities (Incremental Development/Operations):
Architecture-based Scrum of Scrums
Time/Build: 2-4 weeks
Time/Increment: 2-6 months
May 19, 2009
Example: Security kernel; Safety-critical LSI chip
Size, Complexity: Low
Typical Change Rate/Month: 0.3%
Criticality: Extra high
NDI Support: None
Organizational Personnel Capability: Strong formal methods experience
Key Stage I Activities (Incremental Definition): Precise formal
specification
Key Stage II Activities (Incremental Development/Operations):
Formally-based programming language; formal verification
Time/Build: 1-5 days
Time/Increment: 1-4 weeks
Copyright © USC-CSSE
81
University of Southern California
Center for Systems and Software Engineering
Risks Used in Risk Comparison
• Environmental risks
–
–
–
–
Technology uncertainties
Many stakeholders
Complex system-of-systems
Legacy compatibility
• Agility risks
–
–
–
–
–
Scalability
Use of simple design
Personnel turnover
Too-frequent releases
Not enough agile-capable people
• Plan-driven risks
–
–
–
–
Rapid change
Need for rapid results
Emergent requirements
Not enough discipline-capable people
May 19, 2009
Copyright © USC-CSSE
82
University of Southern California
Center for Systems and Software Engineering
SupplyChain.com Home Ground Chart
May 19, 2009
Copyright © USC-CSSE
83
University of Southern California
Center for Systems and Software Engineering
SupplyChain.com Risks
May 19, 2009
Copyright © USC-CSSE
84
University of Southern California
Center for Systems and Software Engineering
SupplyChain.com Mitigations
May 19, 2009
Copyright © USC-CSSE
85
University of Southern California
Center for Systems and Software Engineering
SupplyChain.com Strategy
Startup
1.
Teambuilding
and Shared
Vision
Furnish CRACK
representatives
and alternates
Stakeholders
Project
Manager and
Risk
Management
Team
• Develop benefitsrealization results
chains
• Identify missing
stakeholders
• Enhance mutual
understanding
• Explore goals,
options, technology,
prototypes, risks
2. Systems
Definition and
Architecting
Review;
Commit
to
proceed
Staff and
organize to cover
all successcritical risk areas
Agile Feature Team
May 19, 2009
• Elaborate supply
chain operational
concept, prototypes,
transition strategy
• Evaluate and
determine COTS,
reuse, and
architecture choices
• Prioritize and
sequence desired
capabilities,
outstanding risks
• Establish project
organization, overall
process, and feature
teams
3. Design, Development and Deployment
• Ensure representative
exercise of incremental
capabilities
• Monitor progress and
new operational
development
Review;
Commit
to
proceed
• Monitor project progress,
risk resolution, and new
technology
developments
• Identify and work critical
project-level actions
• Use risk to determine
content of artifacts
• Communicate
proposed
redirections
• Prepare for and
execute
acceptance
tests,
installation,
operational
evolution
• Analyze
feedback on
current
version
• Reprioritize
features
• Prepare
strategy for
next
increment
• Consolidate
process
lessons
learned
• Develop and integrate
new feature increments
LCO
LCA
Copyright © USC-CSSE
86
University of Southern California
Center for Systems and Software Engineering
ICM Summary
• Current processes not well matched to future challenges
– Emergent, rapidly changing requirements
– High assurance of scalable performance and qualities
• Incremental Commitment Model addresses challenges
– Assurance via evidence-based milestone commitment reviews,
stabilized incremental builds with concurrent V&V
• Evidence shortfalls treated as risks
– Adaptability via concurrent agile team handling change traffic and
providing evidence-based rebaselining of next-increment specifications
and plans
– Use of critical success factor principles: stakeholder satisficing,
incremental growth, concurrent engineering, iterative development, riskbased activities and milestones
– Can be adopted incrementally
• Major implications for funding, contracting, career paths
May 19, 2009
Copyright © USC-CSSE
87
University of Southern California
Center for Systems and Software Engineering
Implications for funding, contracting, career paths
• Incremental vs. total funding
– Often with evidence-based competitive downselect
• No one-size-fits all contracting
– Separate instruments for build-to-spec, agile rebaselining, V&V
teams
• With funding and award fees for collaboration, risk management
• Compatible regulations, specifications, and standards
• Compatible acquisition corps education and training
– Generally, schedule/cost/quality as independent variable
• Prioritized feature set as dependent variable
• Multiple career paths
– For people good at build-to-spec, agile rebaselining, V&V
– For people good at all three
• Future program managers and chief engineers
May 19, 2009
Copyright © USC-CSSE
88
University of Southern California
Center for Systems and Software Engineering
ICM Transition Paths
• Existing programs may benefit from some ICM principles and
practices, but not others
• Problem programs may find some ICM practices helpful in
recovering viability
• Primary opportunities for incremental adoption of ICM
principles and practices
– Supplementing traditional requirements and design reviews with
development and review of feasibility evidence
– Stabilized incremental development and concurrent architecture
rebaselining
– Using schedule as independent variable and prioritizing features
to be delivered
– Continuous verification and validation
– Using the process decision table
• See http://csse.usc.edu (tech report 2009-500)
May 19, 2009
Copyright © USC-CSSE
89
University of Southern California
Center for Systems and Software Engineering
References - I
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Beck, K., Extreme Programming Explained, Addison Wesley, 1999.
Boehm, B., “Some Future Trends and Implications for Systems and Software Engineering Processes”,
Systems Engineering 9(1), pp. 1-19, 2006.
Boehm, B., Brown, W., Basili, V., and Turner, R., “Spiral Acquisition of Software-Intensive Systems of
Systems, CrossTalk, Vol. 17, No. 5, pp. 4-9, 2004.
Boehm, B. and Lane J., "21st Century Processes for Acquiring 21st Century Software-Intensive
Systems of Systems." CrossTalk: Vol. 19, No. 5, pp.4-9, 2006.
Boehm, B., and Lane, J., “Using the ICM to Integrate System Acquisition, Systems Engineering, and
Software Engineering,” CrossTalk, October 2007, pp. 4-9.
Boehm, B., and Lane, J., “A Process Decision Table for Integrated Systems and Software
Engineering,” Proceedings, CSER 2008, April 2008.
Boehm, B., “Future Challenges and Rewards for Software Engineers,” DoD Software Tech News,
October 2007, pp. 6-12.
Boehm, B. et al., Software Cost Estimation with COCOMO II, Prentice Hall, 2000.
Boehm, B., Software Engineering Economics, Prentice Hall, 2000.
Carlock, P. and Fenton, R., "System of Systems (SoS) Enterprise Systems for Information-Intensive
Organizations," Systems Engineering, Vol. 4, No. 4, pp. 242-26, 2001.
Carlock, P., and J. Lane, “System of Systems Enterprise Systems Engineering, the Enterprise
Architecture Management Framework, and System of Systems Cost Estimation”, 21st International
Forum on COCOMO and Systems/Software Cost Modeling, 2006.
Checkland, P., Systems Thinking, Systems Practice, Wiley, 1980 (2nd ed., 1999).
Department of Defense (DoD), Defense Acquisition Guidebook, version 1.6, http://akss.dau.mil/dag/,
2006.
Department of Defense (DoD), Instruction 5000.2, Operation of the Defense Acquisition System, May
2003.
Department of Defense (DoD), Systems Engineering Plan Preparation Guide, USD(AT&L), 2004.
Electronic Industries Alliance (1999); EIA Standard 632: Processes for Engineering a System
Galorath, D., and Evans, M., Software Sizing, Estimation, and Risk Management, Auerbach, 2006.
Hall, E.T., Beyond Culture, Anchor Books/Doubleday, 1976.
May 19, 2009
Copyright
©USC-CSSE
© USC-CSSE
90
University of Southern California
Center for Systems and Software Engineering
References -II
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Highsmith, J., Adaptive Software Development, Dorset House, 2000.
International Standards Organization, Information Technology Software Life Cycle Processes, ISO/IEC
12207, 1995
ISO, Systems Engineering – System Life Cycle Processes, ISO/IEC 15288, 2002.
Jensen, R. “An Improved Macrolevel Software Development Resource Estimation Model,”
Proceedings, ISPA 5, April 1983, pp. 88-92.
Krygiel, A., Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33
Lane, J. and Boehm, B., "System of Systems Cost Estimation: Analysis of Lead System Integrator
Engineering Activities", Information Resources Management Journal, Vol. 20, No. 2, pp. 23-32, 2007.
Lane, J. and Valerdi, R., “Synthesizing SoS Concepts for Use in Cost Estimation”, Proceedings of
IEEE Systems, Man, and Cybernetics Conference, 2005.
Lientz, B., and Swanson, E.B., Software Maintenance Management, Addison Wesley, 1980.
Madachy, R., Boehm, B., Lane, J., "Assessing Hybrid Incremental Processes for SISOS Development",
USC CSSE Technical Report USC-CSSE-2006-623, 2006.
Maier, M., “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp
267-284).
Maier, M., “System and Software Architecture Reconciliation,” Systems Engineering 9 (2), 2006, pp.
146-159.
Northrop, L., et al., Ultra-Large-Scale Systems: The Software Challenge of the Future, Software
Engineering Institute, 2006.
Pew, R. W., and Mavor, A. S., Human-System Integration in the System Development Process: A New
Look, National Academy Press, 2007.
Putnam, L., “A General Empirical Solution to the Macro Software Sizing and Estimating Problem,”
IEEE Trans SW Engr., July 1978, pp. 345-361.
Rechtin, E. Systems Architecting, Prentice Hall, 1991.
Schroeder, T., “Integrating Systems and Software Engineering: Observations in Practice,” OSD/USC
Integrating Systems and Software Engineering Workshop,
http://csse.usc.edu/events/2007/CIIForum/pages/program.html, October 2007.
May 19, 2009
Copyright
©USC-CSSE
© USC-CSSE
91
University of Southern California
Center for Systems and Software Engineering
List of Acronyms
B/L
C4ISR
CD
CDR
COTS
DCR
DI
DoD
ECR
EVMS
FCR
FED
FMEA
FRP
GAO
GUI
May 19, 2009
Baselined
Command, Control, Computing, Communications, Intelligence, Surveillance,
Reconnaissance
Concept Development
Critical Design Review
Commercial Off-the-Shelf
Development Commitment Review
Development Increment
Department of Defense
Exploration Commitment Review
Earned Value Management System
Foundations Commitment Review
Feasibility Evidence Description
Failure Modes and Effects Analysis
Full-Rate Production
Government Accountability Office
Graphical User Interface
Copyright
©USC-CSSE
© USC-CSSE
92
University of Southern California
Center for Systems and Software Engineering
List of Acronyms
HMI
HSI
HW
ICM
IOC
IRR
IS&SE
LCO
LRIP
MBASE
NDI
NRC
OC
OCR
OO&D
OODA
O&M
May 19, 2009
(continued)
Human-Machine Interface
Human-System Interface
Hardware
Incremental Commitment Model
Initial Operational Capability
Inception Readiness Review
Integrating Systems and Software Engineering
Life Cycle Objectives
Low-Rate Initial Production
Model-Based Architecting and Software Engineering
Non-Developmental Item
National Research Council
Operational Capability
Operations Commitment Review
Observe, Orient and Decide
Observe, Orient, Decide, Act
Operations and Maintenance
Copyright
©USC-CSSE
© USC-CSSE
93
University of Southern California
Center for Systems and Software Engineering
List of Acronyms
PDR
PM
PR
PRR
RUP
SoS
SoSE
SSE
SW
SwE
SysE
Sys Engr
S&SE
USD (AT&L)
VCR
V&V
WBS
WMI
May 19, 2009
(continued)
Preliminary Design Review
Program Manager
Public Relations
Product Release Review
Rational Unified Process
System of Systems
System of Systems Engineering
Systems and Software Engineering
Software
Software Engineering
Systems Engineering
Systems Engineer
Systems and Software Engineering
Under Secretary of Defense for Acquisition, Technology, and Logistics
Validation Commitment Review
Verification and Validation
Work Breakdown Structure
Warfighter-Machine Interface
Copyright
©USC-CSSE
© USC-CSSE
94
University of Southern California
Center for Systems and Software Engineering
Lifecycle Evolution Motivators as Identified
by CSSE Sponsors and Affiliates
• Current lifecycle models and associated practices
– Based on outdated assumptions
– Don’t always address today’s system development
challenges, especially as systems become more complex
• Need better integration of
–
–
–
–
Systems engineering
Software engineering
Human factors engineering
Specialty engineering (e.g. information assurance, safety)
• Need to better identify and manage critical issues
and risks up front (Young memo)
Additional details found in Backup Charts…
May 19, 2009
Copyright © USC-CSSE
95
University of Southern California
Center for Systems and Software Engineering
Current SysE-SwE Guidance: Outdated Assumptions
Example: SysE Plan Preparation Guide
Sometimes OK for hardware; generally not for software
•
•
•
•
•
•
•
•
•
An omniscient authority pre-establishes the requirements, preferred system
concept, etc. (A4.1, 4.2, 4.4)
Technical solutions are mapped and verified with respect to these (A4.1,
4.2, 4.4)
They and technology do not change very often (A4.2, 4.5, 6.2)
– Emphasis on rigor vs. adaptability
The program has stable and controllable external interfaces (A4.5)
Project organization is function-hierarchical and WBS-oriented (A3.1, 3.2)
All requirements are equally important (A4.2)
Reviews event/product-based vs. evidence-based (A5.2)
The program’s critical path can be defined up front (A6.1)
Can achieve optimum readiness at minimum life-cycle cost (A6.5)
– No slack for contingencies
May 19, 2009
Copyright © USC-CSSE
96
University of Southern California
Center for Systems and Software Engineering
System/Software Architecture Mismatches
- Maier, 2006
• System Hierarchy
• Software Hierarchy
– Part-of relationships;
no shared parts
– Uses relationships;
layered multi-access
– Function-centric; single
data dictionary
– Data-centric; classobject data relations
– Interface dataflows
– Interface protocols;
concurrency challenges
– Dynamic functionalphysical migration
– Static functionalphysical allocation
May 19, 2009
Copyright © USC-CSSE
97
University of Southern California
Center for Systems and Software Engineering
Examples of Architecture Mismatches
• Fractionated, incompatible sensor data management
…
…
Sensor 1
SDMS1
…
Sensor 2
Sensor 3
Sensor n
SDMS2
SDMS3
SDMSn
• “Touch Football” interface definition earned value
– Full earned value taken for defining interface dataflow
– No earned value left for defining interface dynamics
• Joining/leaving network, publish-subscribe, interrupt
handling, security protocols, exception handling, mode
transitions
– Result: all green EVMS turns red in integration
May 19, 2009
Copyright © USC-CSSE
98
University of Southern California
Center for Systems and Software Engineering
Effect of Software Underrepresentation
•Software risks discovered too late
•Slow, buggy change management
•Recent large project reorganization
Original (WBS-based)
New
PM
PM
C4ISR
Sys Engr
C4ISR
Platforms
Software
Sys Engr
SW
Sensors
Networks
WMI
SW
SW
SW
Sensors
SW
May 19, 2009
Copyright © USC-CSSE
Networks
SW
99
University of Southern California
Center for Systems and Software Engineering
ICM Integrates Strengths of Current Process
Models—But not their Weaknesses
• V-Model: Emphasis on early verification and validation
– But not ease of sequential, single-increment interpretation
• Spiral Model: Risk-driven activity prioritization
– But not lack of well-defined in-process milestones
• RUP and MBASE: Concurrent engineering stabilized by
anchor point milestones
– But not software orientation
• Lean Development: Emphasis on value-adding
activities
– But not repeatable manufacturing orientation
• Agile Methods: Adaptability to unexpected change
– But not software orientation, lack of scalability
May 19, 2009
Copyright
©USC-CSSE
© USC-CSSE
100
University of Southern California
Center for Systems and Software Engineering
Case 1: Use NDI
•
•
Exploration phase identifies NDI opportunities
NDI risk/opportunity analysis indicates risks acceptable
–
–
–
–
–
–
•
•
•
•
•
•
•
•
•
Product growth envelope fits within NDI capability
Compatible NDI and product evolution paths
Acceptable NDI volatility, some open-source components highly volatile
Acceptable usability, dependability, interoperability
NDI available or affordable
Example: Small accounting system
Size/complexity: Low
Anticipated change rate (% per month): Low
Criticality: Low
NDI support: Complete
Organizational and personnel capability: NDI-experienced
Key Stage I activities: Acquire NDI
Key Stage II activities: Use NDI
Time/build: Driven by time to initialize/tailor NDI
Time/increment: Driven by NDI upgrades
May 19, 2009
Copyright © USC-CSSE
101
University of Southern California
Center for Systems and Software Engineering
Case 1 Elaboration
•
•
Suppose you have an application for which an appropriate NDI (COTS,
open source, reuse library, customer-furnished package) solution is
available, and other options of developing perhaps a better version
yourself or outsourcing such a development. Even if you produce a
better solution (frequently not the case), you will generally incur more
expense and take longer to begin to capitalize on its benefits. And you
will often find that the NDI package has features that you hadn’t realized
you would need, but are there when you need them.
On the other hand, there are risks that may disqualify some NDIs. They
may be overly complex for your needs, incompatible with your other
applications, or highly volatile. See the discussion in Section 33.1 of
Software Engineering Economics (Boehm, 1981) for more about the pros
and cons of NDI solutions.
May 19, 2009
Copyright © USC-CSSE
102
University of Southern California
Center for Systems and Software Engineering
Case 2: Pure Agile Methods
•
Exploration phase determines
–
–
–
–
–
•
•
•
•
•
•
•
•
•
•
Low product and project size and complexity
Fixing increment defects in next increment acceptable
Existing hardware and NDI support of growth envelope
Sufficient agile-capable personnel
Need to accommodate rapid change, emergent requirements, early user capability
Example: E-services
Size/complexity: Low
Anticipated change rate (% per month): 1-30%
Criticality: Low to medium
NDI support: Good; in place
Organizational and personnel capability: Agile-ready, medium to high
capability
Key Stage I activities: Skip Valuation and Architecting phases
Key Stage II activities: Scrum plus agile methods of choice
Time/build: Daily
Time/increment: 2-6 weeks
May 19, 2009
Copyright © USC-CSSE
103
University of Southern California
Center for Systems and Software Engineering
Case 2 Elaboration
•
•
If your project is small (less than 10 people) and its criticality involves the loss of
discretionary vs. essential funds, a pure agile method such as Extreme Programming
(Beck, 1999), Scrum (Schwaber, 2002), or Crystal (Cockburn, 2002) is generally best if you
have relatively high-capability, agile-ready personnel. The risks of using a more formal,
plan-driven approach are less adaptability to change and belated feedback on the product’s
capabilities. The biggest risk is to try to develop the application all at once for several
months, and then find that it is a mismatch to the users’ needs. Agile developers have
found that the best way to address this risk is to organize the project into short (2-6 week)
delivery increments that may be incomplete, but provide early useful capabilities. Any flaws
can then be detected early and fixed in the next increment (for very high criticality
applications, this would not be acceptable). On a small project it is also easy to set up a
daily build and regression test structure that identifies integration problems early when they
are easier to fix.
Some risks of using the agile approach is that it is harder to write a contract for what is
being developed; that it may sub optimize for early success by using unscalable capabilities
(fourth-generation languages, running all in main memory); or high personnel turnover. See
(Boehm and Turner, 2004), pp. 121-128 for an example risk analysis of an agent-based
event planning system, ending with a decision to go agile.
May 19, 2009
Copyright © USC-CSSE
104
University of Southern California
Center for Systems and Software Engineering
Case 3 Elaboration
•
For medium-size (20-80 people), medium complexity (reasonably mature
and scalable technology; largely compatible shareholders), agile methods
can be scaled using an Architected Agile approach with early investment
in a largely change-prescient architecture and user/developer/customer
team building. For relatively stable projects (0.3-1% change/month),
plan-driven methods can be used with low risk. But for higher rates of
changes (1-10%/month), a more agile approach is less risky. A risk
analysis of a 50-person, medium sized architecture-based agile supply
chain management project is provided on pages 106-121 of (Boehm and
Turner, 2004). A number of organizations in such areas as corporate
infrastructure, medical, aerospace, and ERP applications have reported
significant gains in adaptability and quality of the Architected Agile
approach over plan-driven methods for such projects. However, others
that had less capable and agile-ready people, less management and
customer commitment, and less up-front architecture investment have
not. (Boehm, 2007)
May 19, 2009
Copyright © USC-CSSE
105
University of Southern California
Center for Systems and Software Engineering
Case 4: Formal Methods
•
•
•
•
•
•
•
•
•
•
•
Biggest risks: Software/hardware does not accurately implement
required algorithm precision, security, safety mechanisms, or
critical timing
Example: Security kernel or safety-critical LSI chip
Size/complexity: Low
Anticipated change rate (% per month): 0.3%
Criticality: Extra high
NDI support: None
Organizational and personnel capability: Strong formal methods
experience
Key Stage I activities: Precise formal specification
Key Stage II activities: Formally-based programming language;
formal verification
Time/build: 1-5 days
Time/increment: 1-4 weeks
May 19, 2009
Copyright © USC-CSSE
106
University of Southern California
Center for Systems and Software Engineering
Case 4 Elaboration
•
•
Formal methods involve the development and verification of a precise
mathematical specification of the behavior of a hardware and/of software systems;
an implementation of the system in formal-semantics-based languages; and a
mathematical proof that the implementation is exactly equivalent to the
specification (no more; no less). Such methods are expensive relative to standard
commercial practice and require scarce high-capability personnel, but are
inexpensive relative to a massive product recall, expensive lawsuits, or a massive
loss of valuable and confidential information.
Current formal methods generally require repeating the expensive mathematical
proof process whenever the specification or implementation is changed, making
the approach less viable for systems with highly volatile requirements. In general,
non-developmental items are not precisely specified and verified enough to be
safely used in such systems. Also, formal methods have limited scalability; almost
all fully-verified software systems have less than 10,000 lines of code. However,
some progress is being made toward modularizing such systems so that
implementations and proofs can be built up incrementally via lower-level lemmas
and theorems.
May 19, 2009
Copyright © USC-CSSE
107
University of Southern California
Center for Systems and Software Engineering
Case 5: Hardware Component with
Embedded Software
• Biggest risks: Device recall, lawsuits, production line rework,
hardware-software integration
– DCR carried to Critical Design Review level
– Concurrent hardware-software design
• Criticality makes Agile too risky
– Continuous hardware-software integration
• Initially with simulated hardware
• Low risk of overrun
– Low complexity, stable requirements and NDI
– Little need for risk reserve
– Likely single-supplier software
May 19, 2009
Copyright © USC-CSSE
108
University of Southern California
Center for Systems and Software Engineering
Case 5: Hardware Component with
Embedded Software (continued)
•
•
•
•
•
•
•
•
•
•
Example: Multi-sensor control device
Size/complexity: Low
Anticipated change rate (% per month): 0.3-1%
Criticality: Medium to very high
NDI support: Good, in place
Organizational and personnel capability: Experienced; medium to high
capability
Key Stage I activities: Concurrent hardware and software engineering;
CDR-level ICM DCR
Key Stage II activities: IOC Development, LRIP,FRP, concurrent version
N+1 engineering
Time/build: 1-5 days (software)
Time/increment: Market-driven
May 19, 2009
Copyright © USC-CSSE
109
University of Southern California
Center for Systems and Software Engineering
Case 5 Elaboration
•
The application classes above have been mostly software-intensive. The
differences in economic and risk patterns for hardware-intensive projects (Boehm
and Lane, 2007) will create different risk-based special cases of the ICM. Once a
project commits to a particular manufacturing approach and infrastructure, the
hardware cost of change will be much higher. And the primary costs at risk are in
development and manufacturing, particularly if the component is cheaper to
replace than to fix or if fixes can be accomplished by software workarounds.
Thus, the ICM Stage I activities of producing and validating detailed specifications
and plans are much more cost-effective than for the agile cases above. They will
often go beyond the usual level of detail (Critical Design Review versus evidencebased Preliminary Design Review) for an ICM Development Commitment Review
(DCR), since so many of the details are likely to be major sources of rework
expenditures and delay if wrong. And after Initial Operational Capability (IOC)
development, the rework risks generally dictate an incremental low rate initial
production phase before a full-rate production phase in Stage II. In case the
component is planned to evolve to subsequent hardware-software
reconfigurations, the ICM approach of having a concurrent systems engineering
team developing specifications and plans for the “N+1st “ increment could be
adopted. The length of these later increments will be driven by the product’s
marketplace or competitive situation.
May 19, 2009
Copyright © USC-CSSE
110
University of Southern California
Center for Systems and Software Engineering
Case 6: Indivisible IOC
• Biggest risk: Complexity, NDI uncertainties cause costschedule overrun
– Similar strategies to case 5 for criticality (CDR, concurrent HWSW design, continuous integration)
– Add deferrable software features as risk reserve
• Adopt conservative (90% sure) cost and schedule
• Drop software features to meet cost and schedule
• Strong award fee for features not dropped
– Likely multiple-supplier software makes longer (multi-weekly)
builds more necessary
May 19, 2009
Copyright © USC-CSSE
111
University of Southern California
Center for Systems and Software Engineering
Case 6: Indivisible IOC (continued)
•
•
•
•
•
•
•
•
•
•
Example: Complete vehicle platform
Size/complexity: Medium to high
Anticipated change rate (% per month): 0.3-1%
Criticality: High to very high
NDI support: Some in place
Organizational and personnel capability: Experienced,
medium to high capability
Key Stage I activities: Determine minimum-IOC likely,
conservative cost; Add deferrable software features as risk
reserve
Key Stage II activities: Drop deferrable features to meet
conservative cost; Strong award fee for features not dropped
Time/build: 2-6 weeks (software)
Time/increment: 6-18 months (platform)
May 19, 2009
Copyright © USC-CSSE
112
University of Southern California
Center for Systems and Software Engineering
Case 6 Elaboration
•
More complex hardware intensive systems, such as aircraft, may have a great deal of software
that can be incrementally developed and tested, but may have an indivisible hardware IOC that
must be developed as a unit before it can be safely tested (e.g., an automobile or aircraft’s
braking system or full set of safety-critical vehicle controls). Relative to the cost and duration of
the software increments, the indivisible hardware IOC has two significant sources of risk:
–
–
–
•
It cannot fit into smaller software increments
It cannot drop required hardware features to meet IOC schedule/cost/quality as independent
variable.
The first can be addressed by synchronizing the hardware IOC with the Nth software increment
(e.g., (Rechtin and Maier, 1997) osculating orbits for hardware and software).
If some combination of schedule/cost/quality is truly the project IOC’s independent variable, then
one does not want to commit to a most-likely combination of schedule/cost/quality, as there will
be a roughly 50% chance of an overrun or quality shortfall in completing the indivisible IOC.
Rather, it is better to determine a conservative IOC cost and schedule for meeting the quality
objective, and use the difference between the conservative cost and schedule and the mostlikely cost and schedule as a risk reserve. The best way to do this is to use the conservative cost
and schedule as the IOC target, and to determine a set of desired but non-essential, easy to drop
software features that could be developed within the risk reserve. Then, if the most-likely
indivisible IOC capability begins to overrun, some desired software features can be dropped
without missing the scheduled delivery time and cost. There should also be a strong award-fee
incentive for the developers to minimize the number of features that need to be dropped.
May 19, 2009
Copyright © USC-CSSE
113
University of Southern California
Center for Systems and Software Engineering
Case 7 Elaboration
•
•
Our experiences in developing USC web-service applications between 1996 and 2004 was that
they went from 28% of the application’s functionality being delivered by NDI components to 80%
(Yang et al, 2005). A similar trend was identified by the 2001 Standish Report, which reported
that 53% of the functionality of commercial software applications was being delivered by NDI
components in 2000 (Standish, 2001). The economics of NDI-intensive systems dictates a
bottom-up versus a top-down approach to system development, in which the capability envelope
of the NDI determines the affordable requirements, rather than a top-down requirements-tocapability approach. A large supply-chain management system may need to choose among
several NDI candidates each for such functions as inventory control, trend analysis,
supplier/customer relations management, transportation planning, manufacturing control, and
financial transactions; and evaluate not only the candidates’ cost/performance aspects, but also
their interoperability with each other and with the corporation’s legacy infrastructure. Besides
NDI assessment, other significant sources of effort can be NDI tailoring, NDI integration, and
effect of NDI version volatility and obsolescence; see (Yang et al, 2005).
A particular challenge in Stage II is the effect of NDI volatility and obsolescence. Surveys have
indicated that commercial NDI products have a new release about every 10 months, and that old
releases are supported by the vendor for about 3 releases. Some large systems have had about
120 NDI components, indicating that about 12 components will have new releases each month,
and that not upgrading will leave each component unsupported in about 30 months. In such
cases, a great deal of attention needs to be paid to upgrade synchronization, and to pro-active
NDI evolution influencing. Some large organizations synchronize their NDI upgrades to their
major re-training cycles of about 12-18 months. For additional NDI best practices, see (Wallnau
et al, 2002).
May 19, 2009
Copyright © USC-CSSE
114
University of Southern California
Center for Systems and Software Engineering
Case 8: Hybrid Agile/Plan-Driven System
•
•
•
•
•
•
•
•
•
•
•
Biggest risks: large scale, high complexity, rapid change, mixed
high/low criticality, partial NDI support, mixed personnel capability
Example: C4ISR system
Size/complexity: Medium to very high
Anticipated change rate (% per month): Mixed parts; 1-10%
Criticality: Mixed parts; medium to very high
NDI support: Mixed parts
Organizational and personnel capability: Mixed parts
Key Stage I activities: Full ICM; encapsulated agile in high
changed; low-medium criticality parts (often HMI, external
interfaces)
Key Stage II activities: Full ICM, three-team incremental
development, concurrent V&V, next-increment rebaselining
Time/build: 1-2 months
Time/increment: 9-18 months
May 19, 2009
Copyright © USC-CSSE
115
University of Southern California
Center for Systems and Software Engineering
Case 8 Elaboration
•
Some large, user-intensive systems such as C4ISR systems, air traffic control
systems, and network control systems have a mix of relatively stable, highcriticality elements (sensors and communications; key business logic) and more
volatile, more moderately critical elements such as GUI displays, electronic
warfare countermeasures, and interfaces with externally evolving systems. As
many acquisitions of this nature have shown, it is highly risky to apply one-sizefits-all processes and incentive structures to these differing classes of elements.
Instead, in Stage I, it is important to determine which system elements belong in
which class along with the other functions of understanding needs, envisioning
opportunities, NDI assessment, scoping the system, and determining feasible
requirements and increments performed for complex systems in Stage I; to
architect the system to encapsulate the volatile parts for development by agile
teams; and to organize to use plan-driven development for the other elements of
the overall system. In Stage II, the cycles for the agile teams are likely to be
significantly shorter than those for the plan-driven teams.
May 19, 2009
Copyright © USC-CSSE
116
University of Southern California
Center for Systems and Software Engineering
Case 9: Multi-Owner System of Systems
•
Biggest risks: all those of Case 8 plus
– Need to synchronize, integrate separately-managed, independently-evolving
systems
– Extremely large-scale; deep supplier hierarchies
– Rapid adaptation to change extremely difficult
•
•
•
•
•
•
•
•
•
Example: Net-centric military operations or global supply chain
management
Size/complexity: Very high
Anticipated change rate (% per month): Mixed parts; 1-10%
Criticality: Very high
NDI support: Many NDIs; some in place
Organizational and personnel capability: Related experience, medium to
high
Key Stage I activities: Full ICM; extensive multi-owner teambuilding,
negotiation
Key Stage II activities: Full ICM; large ongoing system/software
engineering effort
Time/build: 2-4 months
Time/increment:18-24 months
May 19, 2009
Copyright © USC-CSSE
117
University of Southern California
Center for Systems and Software Engineering
Case 9 Elaboration
•
•
•
In this situation, your goal is to integrate a set of existing systems (or guide and evolve the
integration of a set of existing systems). These systems are primarily developed, owned, and
maintained by an organization other than the one that is attempting to manage and guide the set
of systems as a system of systems. Because of the independence of these constituent systems,
the SoS organization has little or no formal control over the processes used to maintain and
evolve the constituent systems. The SoS may be an enterprise wide business SoS, with the
constituents being primarily COTS products along with some legacy applications. Or it may be
Department of Defense (DoD) warfighting SoS, where the constituent legacy systems are
integrated to increase capabilities on the battle field.
Traditional systems engineering (SE) activities are typically tailored for the SoS case to define an
SoS architecture, better coordinate the activities of multiple systems in migrating to the SoS
architecture, and provide synchronization points for the SoS. In (OUSD AT&L, 2008), pilot
studies have shown that many DoD SoS have re-organized the traditional SE activities into a set
of seven core elements: 1) translating capability objectives, 2) understanding systems and
relationships, 3) assessing performance to capability objectives, 4) developing, evolving, and
maintaining SoS design, 5) monitoring and assessing changes, 6) addressing new requirements
and options, and 7) orchestrating updates to SoS. Further analysis shows, that these elements
map fairly well to the hybrid agile/plan-driven case (Case 8) at the SoS level.
What makes this case different from Case 8, is that each of the constituent systems is using their
own processes, which could be any of the above cases, depending on the scope, complexity,
and characteristics of the constituent system. What is key is that the Case 9 extends Case 8 to
include information or participation of the constituent systems in the agile planning activities and
lets the “battle rhythm” of the constituent system increments guide the SoS plan-driven and V&V
activities.
May 19, 2009
Copyright © USC-CSSE
118
University of Southern California
Center for Systems and Software Engineering
Case 10: Family of Systems
•
Biggest risks: all those of Case 8 plus
– Need to synchronize, integrate separately-managed, independentlyevolving systems
– Extremely large-scale; deep supplier hierarchies
– Rapid adaptation to change extremely difficult
•
•
•
•
•
•
•
•
•
Example: Medical device product line
Size/complexity: Medium to very high
Anticipated change rate (% per month): 1-3%
Criticality: Medium to very high
NDI support: Some in place
Organizational and personnel capability: Related experience,
medium to high capability
Key Stage I activities: Full ICM; full stakeholder participation in
product line scoping; strong business case
Key Stage II activities: Full ICM; extra resources for first system,
version control, multi-stakeholder support
Time/build: 1-2 months
Time/increment: 9-18 months
May 19, 2009
Copyright © USC-CSSE
119
University of Southern California
Center for Systems and Software Engineering
Case 10 Elaboration
•
Families of systems are typically a set of systems that belong to a product line and
can be easily used to customize a solution for a given need. This might be a suite
of medical devices or a suite of applications to customer support. This is often the
set of systems developed by a vendor that become the NDI components for CASE
7 above. Again, the rigor required for the SoS case is present here. However, in
this situation, the family of systems is typically owned and evolved by a single
organization/vendor and presents a case where the owning organization has much
more control over the evolution of the components of the family of systems, thus
possibly reducing some risks and allowing the ICM process to be a little more
streamlined.
May 19, 2009
Copyright © USC-CSSE
120
University of Southern California
Center for Systems and Software Engineering
Case 11 Elaboration
•
•
•
•
•
A Brownfield counterexample involved a major US corporation that used a Greenfield systems engineering and
development approach to develop a new Central Corporate Financial System to replace a patched-together collection of
strongly-coupled and poorly documented COBOL business data processing programs. The system included an early
Enterprise Resource Planning (ERP) system that was well matched to the corporation's overall financial needs, but not to
its detailed business processes, which included various workarounds to compensate for difficulties with the legacy
software. The new system was well organized to support incremental implementation, but was dropped at a cost of $40
million after two failed tries to provide continuity of service, due primarily to the infeasibility of incrementally phasing out
the legacy software and business processes compatibly with the new system's incremental capabilities.
The ICM can be used to organize systems engineering and acquisition processes in ways that better support Brownfield
legacy software re-engineering so that organizations can better provide continuity of services. In particular, its
concurrent activities of Understanding Needs, Envisioning Opportunities, System Scoping and Architecting, Feasibility
Evidence Development, and Risk/Opportunity Assessment enable projects to focus specifically on their legacy system
constraints and on opportunities to deal with them.
The application of the ICM to the Brownfield corporate counterexample situation would have avoided the failures of the
Greenfield development. Its Understanding Needs activity would have determined the ways in which the legacy system
had intertwined financial and other business services. For examples, Project Services included budgeting and
scheduling, work breakdown system accounting, earned value management intertwined with requirements, version and
configuration management; Contract Services included expenditure category management, billing, and receivables
management intertwined with deliverables management and engineering change proposal tracking; with similar
intertwining in Personnel Services and Marketing Services.
The ICM's Envisioning Opportunities activity would have identified opportunities to use large-scale refactoring methods to
decouple the financial and non-financial elements of these services in ways that would make it feasible for them to
interoperate with a Financial Services element. Its System Scoping and Architecting activity would have repackaged the
legacy software and developed the new architecture around financial services that would support incremental phaseout,
and its Feasibility Evidence Development activity would have assessed any outstanding risks and covered them with risk
management plans that would be tracked to ensure project success.
A good example of a Brownfield methodology with several case study examples is provided in (Hopkins and Jenkins,
2008).
May 19, 2009
Copyright © USC-CSSE
121
University of Southern California
Center for Systems and Software Engineering
Case 12a: Net-Centric Services – Community Support
•
•
•
•
•
•
•
•
•
•
Example: Community services or special interest group
Size/complexity: Low to medium
Anticipated change rate (% per month): 0.3-3
Criticality: Low to medium
NDI support: Tailorable service elements
Organizational and personnel capability: NDI-experienced
Key Stage I activities: Filter, select, compose, tailor NDI
Key Stage II activities: Evolve tailoring to meet community needs
Time/build: <= 1 day
Time/increment: 2-12 months
May 19, 2009
Copyright © USC-CSSE
122
University of Southern California
Center for Systems and Software Engineering
Case 12a Elaboration
•
•
Net-Centric Services for community support tend to fall into two categories. One
involves community service organizations providing needed services for child care,
elder care, handicapped, homeless, or jobless people. Their information
processing service needs include tracking of clients, volunteers, donors,
donations, events, and provision of news and communication services. These
used to require considerable programming to realize, but now can be realized by
combining and tailoring NDI packages offering such services. The other category
of net-centric services for community support involves special interest groups
(professionals, hobbyists, committees, ethnic groups) with similar needs for
tracking members, interests, subgroups, events, and provision of news and
communication services. Development of such capabilities involves much more
prototyping and much less documentation than is needed for more programmingoriented applications; for example, the internals of the NDI packages are generally
unavailable for architectural documentation, and the resulting system capabilities
are more driven by NDI capabilities than by prespecified requirements documents.
A good example of how the ICM provides support for such net-centric services is
provided in (Boehm and Bhuta, 2008) and the entire journal provides additional
approaches and case studies of net-centric services development.
May 19, 2009
Copyright © USC-CSSE
123
University of Southern California
Center for Systems and Software Engineering
Case 12b: Net-Centric Services – Quick Response
Decision Support
•
•
•
•
•
•
•
•
•
•
Example: Response to competitor initiative
Size/complexity: Medium to high
Anticipated change rate (% per month): 3-30
Criticality: Medium to high
NDI support: Tailorable service elements
Organizational and personnel capability: NDI-experienced
Key Stage I activities: Filter, select, compose, tailor NDI
Key Stage II activities: Satisfy quick response; evolve or phase out
Time/build: <= 1 day
Time/increment: Quick response-driven
May 19, 2009
Copyright © USC-CSSE
124
University of Southern California
Center for Systems and Software Engineering
Case 12b Elaboration
•
Another form of net-centric services involves rapidly configuring a
capability to analyze alternative decision options in response to a
competitor's initiative. These might involve a competitor beginning to
penetrate a business's home territory, and might involve the need to
evaluate the relative costs and benefits of alternative strategies of
expanding services, expanding service locations, or repricing products or
services. NDI packages to help analyze geographical, logistical, human
resources, and financial implications of alternative competitive response
decisions can be rapidly configured to provide a quick-response capability
for converging on a decision. Once the decision is made, the resulting
capability may be phased out, or evolved into a more general and
maintainable capability for similar future situations. Again, the
November/December 2008 special issue of IEEE Software on "Opportunistic
Software Systems Development" provides additional perspectives on this
area of net-centric services development.
May 19, 2009
Copyright © USC-CSSE
125