Guide for Using the Incremental Commitment Model (ICM) for
Systems Engineering of DoD Projects
Version 0.5
Barry Boehm
Jo Ann Lane
December 31, 2008
106753919
i
Version Date: 12/31/08
Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects
Version 0.5
Version History
Date
Author
Version
Changes made
Rationale
12/31/08
BB, JAL
0.5
 First draft created.
 Create first draft.
106753919
ii
Version Date: 12/31/08
Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects
Version 0.5
Table of Contents
GUIDE FOR USING THE INCREMENTAL COMMITMENT MODEL (ICM) FOR
SYSTEMS ENGINEERING OF DOD PROJECTS............................................................ I
VERSION 0.5 ................................................................................................................... I
VERSION HISTORY ...................................................................................................... II
TABLE OF CONTENTS ................................................................................................. III
TABLE OF TABLES ..................................................................................................... VII
TABLE OF FIGURES .................................................................................................. VIII
0 Executive Summary....................................................................................................... 1
0.1
Abstract ............................................................................................................ 1
0.2
Purpose and Scope ............................................................................................ 1
0.3
Motivation and Context: Current DoD Needs and Future Challenges ................. 2
0.4
The 6 Key Systems Engineering Process Principles ............................................ 4
0.5
Overview of the Generic ICM and DoD Life Cycle Processes ............................. 5
0.6
ICM Metaphor and Competitive Prototyping Example ...................................... 9
0.7
The DoD Version of the ICM ........................................................................... 11
0.8
What is Being Concurrently Engineered in the ICM? ...................................... 13
0.9
How is All This Concurrent Engineering Synchronized and Stabilized? ........... 15
0.10 Project Experience with ICM Principles .......................................................... 17
0.11 Structure and Content of the Guide ................................................................. 18
0.12 Conclusions ..................................................................................................... 21
0.13 References....................................................................................................... 21
1 Chapter 1: ICM Fundamentals – Fundamental Principles ............................................ 24
1.1
Introduction .................................................................................................... 24
1.2 Key Principle #1: Commitment and Accountability of System Engineers,
Developers, and Managers ........................................................................................ 24
1.3
Key Principle #2: Stakeholder Satisficing......................................................... 24
1.4 Key Principle #3: Incremental and Evolutionary Growth of System Definition
and Stakeholder Commitment .................................................................................. 24
106753919
iii
Version Date: 12/31/08
Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects
Version 0.5
1.5
Key Principle #4: Iterative System Development and Definition ....................... 25
1.6
Key Principle #5: Concurrent System Definition and Development .................. 25
1.7
Key Principle #6: Evidence-Based, Risk Driven Decision Milestones ................ 25
1.8
References....................................................................................................... 25
2 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs ..................................... 26
2.1 Developing and Using Feasibility Evidence for Life Cycle Commitment
Milestones ................................................................................................................ 26
2.2
Nature of FEDs and AP Milestones .................................................................. 26
2.3
Problems Encountered without FED................................................................ 27
2.4
Problems Avoidable with FED ......................................................................... 28
2.5
AT&T Experience with AP Reviews ................................................................ 29
2.6
Overview of ICM Phases, Anchor Points, and Decision Options ....................... 30
2.7
Key Point: Need to Show Evidence ................................................................. 31
2.8
Off-Nominal Architecture-Breakers ................................................................ 31
2.9
Common Examples of Inadequate Evidence..................................................... 32
2.10 Examples of Making the Evidence Adequate .................................................... 33
2.11 ICM Detailed Activity Descriptions ................................................................. 33
2.12 Focus of Each Commitment Review ................................................................. 36
2.13 Lean Risk Management Plan ........................................................................... 36
2.14 FED Development Process Framework ............................................................ 37
2.15 Steps for Developing Feasibility Evidence ........................................................ 38
2.16 Large-Scale Simulation and Testbed FED Preparation Example ...................... 40
2.17 Example of FED Risk Evaluation Criteria ....................................................... 41
2.18 Conclusions ..................................................................................................... 41
2.19 References....................................................................................................... 41
3 Chapter 3: ICM Fundamentals – Incremental Adoptability.......................................... 43
3.1
Introduction .................................................................................................... 43
3.2 Supplementing Traditional Requirements and Design Reviews with Development
and Review of Feasibility Evidence............................................................................ 43
3.3
Stabilized Incremental Development and Concurrent Architecture Rebaselining
43
3.4 Using Schedule as Independent Variable and Prioritizing Features to be
Delivered .................................................................................................................. 43
3.5
106753919
Continuous Verification and Validation ........................................................... 43
iv
Version Date: 12/31/08
Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects
3.6
Version 0.5
Using the Process Decision Table ..................................................................... 43
4 Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration
Phase ............................................................................................................................. 44
4.1
Introduction .................................................................................................... 44
4.2
Phase Key Elements ........................................................................................ 46
4.3
Goals and Objectives ....................................................................................... 48
4.4
Entry Conditions ............................................................................................. 48
4.5
Inputs ............................................................................................................. 48
4.6
Steps (Concurrent) .......................................................................................... 48
4.7
Exit Conditions ............................................................................................... 49
4.8
Work Products ................................................................................................ 49
4.9
Pitfalls ............................................................................................................ 50
5 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table ............. 51
5.1
Introduction .................................................................................................... 51
5.2
The ICM Process Decision Table ..................................................................... 51
5.3
Using the Process Decision Table ..................................................................... 55
5.4
References....................................................................................................... 61
6 Chapter 6: Stage I: Incremental Definition – The Materiel Solution Analysis &
AoA/Valuation Phase ..................................................................................................... 62
6.1
Introduction .................................................................................................... 62
6.2
Phase Key Elements ........................................................................................ 62
6.3
Goals and Objectives ....................................................................................... 64
6.4
Entry Conditions ............................................................................................. 64
6.5
Inputs ............................................................................................................. 64
6.6
Steps (Concurrent) .......................................................................................... 64
6.7
Exit Conditions ............................................................................................... 65
6.8
Work Products ................................................................................................ 65
6.9
Pitfalls ............................................................................................................ 65
7 Chapter 7: Stage I: Incremental Definition – The Technology Development &
CDD/Foundations Phase ................................................................................................ 67
7.1
Introduction .................................................................................................... 67
7.2
Phase Key Elements ........................................................................................ 69
7.3
Goals and Objectives ....................................................................................... 71
106753919
v
Version Date: 12/31/08
Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects
Version 0.5
7.4
Entry Conditions ............................................................................................. 71
7.5
Inputs ............................................................................................................. 71
7.6
Steps (Concurrent) .......................................................................................... 71
7.7
Exit Conditions ............................................................................................... 72
7.8
Work Products ................................................................................................ 72
7.9
Pitfalls ............................................................................................................ 73
8 Chapter 8: Stage II: Incremental Developments – Engineering and Manufacturing
Development and Beyond ............................................................................................... 74
8.1
Introduction .................................................................................................... 74
8.2
Phase Key Elements ........................................................................................ 78
8.3
Goals and Objectives ....................................................................................... 80
8.4
Entry Conditions ............................................................................................. 80
8.5
Inputs ............................................................................................................. 80
8.6
Steps (Concurrent) .......................................................................................... 80
8.7
Exit Conditions ............................................................................................... 81
8.8
Work Products ................................................................................................ 81
8.9
Pitfalls ............................................................................................................ 82
8.10 Special Cases ................................................................................................... 82
106753919
vi
Version Date: 12/31/08
Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects
Version 0.5
Table of Tables
Table 0-1. Number of Top-5 Projects Explicitly Using ICM Principles....................................... 17
Table 2-1. Steps for Developing FED ........................................................................................... 39
Table 4-1. Needs and Opportunities (Exploration) ...................................................................... 47
Table 5-1. Characteristics of the Risk-Driven Special Cases of the ICM .................................... 53
Table 5-2. Key Activities for Each Special Case of the ICM ........................................................ 54
Table 6-1 Materiel Solution Analysis; Analysis of Alternatives (Valuation) ................................ 63
Table 7-1. Technology Development and Capability Development Document: TD-CDD
(Foundations)................................................................................................................................ 70
Table 8-1. Engineering and Manufacturing Developmentn (Developmentn) ................................ 79
106753919
vii
Version Date: 12/31/08
Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects
Version 0.5
Table of Figures
Figure 0-1. The Defense Acquisition Management System as in DODI 5000.02........................... 6
Figure 0-2. Overview of the Generic Incremental Commitment Life Cycle Process. .................... 7
Figure 0-3. DoD Version of ICM for Evolutionary Deveolpment ................................................ 12
Figure 0-4. Requirements and Acquisition Flow as in DoDI 5000.02 ......................................... 13
Figure 0-5. ICM Activity Categories and Level of Effort ............................................................. 14
Figure 0-6. Feasibility Evidence Description Content ................................................................. 15
Figure 0-7. An Example Combining Directed SoS Engineering and Component Supplier
Processes using ICM Anchor Point Reviews. ............................................................................... 16
Figure 2-1. Problems Encountered without FED: 15-Month Architecture Rework Delay .......... 28
Figure 2-2. AT&T Experience with AP Reviews .......................................................................... 30
Figure 2-3. Critical off-Nominal Condition that Cannot Be Handled by the Nominal-Case
Architecture................................................................................................................................... 32
Figure 2-4. ICM Detailed Activity Descriptions........................................................................... 35
Figure 2-6. Large-Scale Simulation and Testbed FED Preparation Example ............................ 40
Figure 4-1. Overview of the ICM - Exploration Highlighted ....................................................... 45
Figure 4-2. ICM Activity Categories and Level of Effort ............................................................. 46
Figure 6-1. Overview of the ICM – MSA and AoA Highlighted ................................................... 62
Figure 7-1. Overview of the ICM – Technology Development and CDD Highlighted ................ 68
Figure 7-2. Categorization of DoD Program Assessment Negative Findings ............................. 69
Figure 8-1. Overview of the Generic Incremental Commitment Life Cycle Process – Stage II
Highlighted ................................................................................................................................... 74
Figure 8-2. ICM Stage II: Increment Activities ............................................................................ 76
Figure 8-3. DoD Version of ICM for Evolutionary Development ................................................ 78
Figure 8-4. Example of SoSE Coordination Challenge ................................................................ 82
106753919
viii
Version Date: 12/31/08
Executive Summary
Version 0.5
0Executive Summary
0.1
Abstract
Future mission-critical DoD combat platforms, systems of systems, and network-centric services
will have many usage uncertainties and emergent characteristics. Their hardware, software, and
human factors will need to be concurrently engineered, risk-managed, and evolutionarily
developed to converge on cost-effective system operations and mission success.
The recent revision of DoDI 5000.02 provides a much improved policy for dealing with these
future challenges, including its starting with Needs and Opportunities, its emphasis on early
Analysis of Alternatives as an entry condition for Milestone A, its inclusion of a passed
Preliminary Design Review as an entry condition for Milestone B, and its emphasis on
evolutionary versus single-increment or prespecified increments of development.
These emphases on DoDI 5000.02 are also key emphases in the Incremental Commitment Model
(ICM), along with more detailed guidance on how to apply them. The ICM has been developed
based on best commercial practices for addressing such future challenges; on discussions with
contributors to DoDI 5000.02; on successful use of the ICM principles on previous DoD
projects; and on its exploratory use on highly complex, future-precursor, net-centric DoD
systems of systems.
The ICM builds on the experience-based critical success factor principles of stakeholder
satisficing, incremental definition, iterative evolutionary system growth, concurrent engineering,
and evidence-based, risk-driven management milestones. It is not a one-size-fits-all process
model, but has risk-driven options at each milestone decision point to enable projects to adapt to
their particular situations. It provides a decision table for early determination of common specialcase life cycle processes to avoid overkill in project ceremony and documentation.
It also incorporates the strengths of existing V, concurrent engineering, spiral, agile, and lean
process models. As with these other models, it is not fully tested for its ability to support DoDI
5000.02 in all project situations, but it has been formulated to make it straightforward to do so.
0.2
Purpose and Scope
The purpose of this Guide is to enable Department of Defense (DoD) projects to better cope with
their current and emerging systems engineering challenges, in the context of the recently-revised
DoD Instruction 5000.02, “Operation of the Defense Acquisition System.” The definition of the
Incremental Commitment Model (ICM) has been co-evolving with the redefinition of the
previous DoDI 5000.2. This began with a series of work sessions in 2005 with OUSD (AT&L)
personnel to clarify the nature of “spiral development” as recommended in the previous DoDI
5000.2. The redefinitions leading to the ICM were focused both on avoiding common
misinterpretations of “Spiral development” and on stimulating systems engineering and
acquisition practices better tuned to DoD systems of the future rather than to systems of the past.
DoD projects are increasingly encountering “tipping points” in the way their new systems are
acquired, the way that existing systems are modified and integrated to form systems of systems
(SoSs), and the way that systems that are part of SoSs are modified to contribute to new SoS
106753919
-1-
Version Date: 12/31/08
Executive Summary
Version 0.5
capabilities. Traditionally, a DoD system’s requirements could be specified in advance, and used
as a basis for an often fixed-price, build-to-specification contract to develop given numbers of
the systems to be delivered.
Several trends have made this approach increasingly obsolete, but the use of traditional DoD
systems acquisition and engineering guidance has still been the path of least resistance for DoD
program managers. This has become a major root cause for the shortfalls and overruns on recent
DoD projects.
Recent DoD initiatives besides the revision of DoD 5000.02 such as the Competitive Prototyping
initiative and the DoD Systems of Systems Engineering Guide, are beginning to provide the next
generation of guidance for DoD systems acquisition. This guidebook is intended to be part of
this next generation of guidance for systems engineering of DoD projects.
The Incremental Commitment Model (ICM) presented in the Guide is a synthesis of best
commercial systems engineering practices and experiences with large DoD space mission
systems and precursor SoSs such as the Army’s Future Combat Systems program. The ICM’s
overall structure and underlying principles were largely developed as part of a DoD-sponsored
National Research Council (NRC) study on Human–System Integration in the System
Development Process.
The NRC study’s process element involved analyzing the strengths and shortfalls of current
systems engineering-related models, and synthesizing a process framework and decision
guidelines that capitalized on their strengths while avoiding their shortfalls. The resulting ICM
has been used and evolved on systems ranging from small-business and public-service
applications to portions of very large DoD SoSs.
The full ICM has a number of implications for how future DoD systems would be defined,
financed, acquired, and evolved. This Guide identifies these implications, but addressing them is
beyond its scope.
However, the Guide does show how to selectively apply key ICM practices to strengthen
projects being acquired under traditional methods. It also includes a Decision Table that
identifies conditions under which traditional DoD systems acquisition and engineering practices
remain applicable.
The remainder of this Executive Summary provides the motivation and context for using the
ICM; a summary of its key distinguishing features and principles; and its relationships to DoDrelevant policies and standards. As DoD policy and guidance becomes more definitive and as
improvements over the draft material are incorporated, the Guide will evolve into a definitive
version.
0.3
Motivation and Context: Current DoD Needs and Future Challenges
Current DoD systems challenges that are driving the need for better systems engineering and
acquisition processes include:

Complex, multi-owner, net-centric SoS. Although these are complex, they are key to
avoiding the shortfalls of current collections of incompatible, separately-developed,
stovepiped systems. These shortfalls cause numerous operational deficiencies such as
unacceptable delays in service, uncoordinated and conflicting plans, ineffective or dangerous
106753919
-2-
Version Date: 12/31/08
Executive Summary
Version 0.5
decisions, and inabilities to cope with fast-moving events. Multiple owners of key
interdependent systems make synchronization and stabilization of independently evolving
systems a major challenge [1]. The ICM’s anchor point milestones have been developed to
facilitate such synchronization and stabilization.

Emergent requirements. The most appropriate user interfaces and collaboration modes for a
complex human-intensive system are not specifiable in advance, but emerge with system
prototyping and usage. Forcing them to be prematurely and precisely specified generally
leads to poor business or mission performance and expensive late rework and delays. The
ICM provides support for incremental and concurrent definition of system requirements and
solutions, including competitive prototyping approaches [2].

Rapid change. Specifying current-point-in-time snapshot requirements on a cost-competitive
contract generally leads to a big design up front, and a point-solution architecture that is hard
to adapt to new developments. Each of the many subsequent changes then leads to
considerable nonproductive rework of documents and software (or worse, of hardware), and
in renegotiating contracts [3]. The ICM’s incremental definition and development stages
directly support the recommendations in [3] for shorter increments and evolutionary
development.

Reused components and legacy system constraints. Much of current DoD systems
engineering and acquisition guidance is oriented around Greenfield development of new
systems. However, these systems are increasingly challenged by Brownfield legacy system
constraints [19]. Building all of one’s own components from scratch is economically
infeasible for complex systems. And at the same time, reuse-based development has major
bottom-up development implications that can be incompatible with pure top-down
requirements-first approaches. Prematurely specifying requirements (as one project
example, hasty specification of a 1-second response time requirement when later prototyping
showed that 4 seconds would be acceptable) that disqualify otherwise most cost-effective
reusable components often leads to overly expensive, late, and unsatisfactory systems [4].
The ICM includes guidance for reuse-driven and legacy-driven system definition and
development.

High assurance of qualities. Today’s systems need higher assurance levels of such qualities
as safety, security, reliability/availability/maintainability, performance, adaptability,
interoperability, usability, and scalability. Just assuring one of these qualities for a complex
SoS is difficult. Given the need to “satisfice” (not everybody gets everything they want, but
everybody gets something they are satisfied with) among multiple system owners with
different quality priorities, complex sources of conflict and tradeoff relationships make
multi-attribute satisficing even more challenging [5]. The ICM’s principles of stakeholder
satisficing and evidence-based commitment milestones help ensure that key stakeholders’
primary quality concerns are addressed.

Increasing importance of software and human factors. Rapid change and emergent
requirements imply the need to have rapidly adaptable systems, increasing the importance of
software’s flexibility and electronic upgradeability. Increasing SoS complexity presents
significant challenges to warfighters’ and commanders’ situation-understanding capabilities,
increasing the importance of human engineering. Such concerns led to one of the top
recommendations from the October 2006 OUSD (AT&L) Defense Software Strategy
106753919
-3-
Version Date: 12/31/08
Executive Summary
Version 0.5
Summit: to find ways of better integrating software engineering into the systems engineering
and acquisition process [6]. Concurrently, the NRC study was addressing the problem of
better integrating human factors into the systems engineering and acquisition process
[7]. One of its results was to refine and recommend the ICM and its principles as a process
framework for integrating hardware, software, and human factors engineering into the
systems engineering process.
0.4
The 6 Key Systems Engineering Process Principles
The NRC human-systems integration study analyzed the strengths and difficulties of current
process models in addressing these challenges. Each had strengths, but needed further
refinements to address all of the six challenges above. The most important conclusion, though,
was that there were key process principles that address the challenges, and that forms of the
models were less important than their ability to adopt the principles. These key principles are:
1. Commitment and accountability of system engineers, developers, and managers. Without key
personnel commitment and accountability for the system under development, there is no way
to build trust among the system’s stakeholders. It is too easy to overpromise and depart. As
recommended in the NRC Kaminski report [3], development increments must be short
enough that the key program leaders remain engaged in their current jobs. And there must be
clear visibility of progress versus plans up and down the supplier chain.
2. Stakeholder satisficing. If a system development process presents a success-critical
operational or development stakeholder with the prospect of an unsatisfactory outcome, the
stakeholder will generally refuse to cooperate, resulting in an unsuccessful system.
Stakeholder satisficing includes identifying the success-critical stakeholders and their value
propositions; negotiating a mutually satisfactory set of system requirements, solutions, and
plans; and managing proposed changes to preserve a mutually satisfactory outcome.
3. Incremental and evolutionary growth of system definition and stakeholder commitment. This
characteristic captures the often incremental discovery of emergent requirements and
solutions via methods such as prototyping, operational exercises, and use of early system
capabilities. Requirements and commitment cannot be monolithic or fully pre-specifiable for
complex, human-intensive systems; increasingly detailed understanding, trust, system
definition, and commitment is achieved through an evolutionary process.
4. Iterative system development and definition. The incremental and evolutionary approaches
lead to cyclic refinements of requirements, solutions, and development plans. Such iteration
helps projects to learn early and efficiently about operational and quality requirements and
priorities.
5. Concurrent system definition and development. Initially, this includes concurrent engineering
of requirements and solutions, and integrated product and process definition. In later ICM
increments, change-driven rework and rebaselining of next-increment requirements,
solutions, and plans occurs concurrently with stabilized development of the current system
increment. This allows early fielding of core capabilities, continual adaptation to change,
and timely growth of complex systems without waiting for every requirement and subsystem
to be defined.
106753919
-4-
Version Date: 12/31/08
Executive Summary
Version 0.5
6. Evidence-based, risk driven decision milestones. The key to synchronizing and stabilizing all
of this concurrent activity is a set of evidence-based, risk-driven stakeholder decision
milestones. At these milestones, evidence of the business, technical, and operational
feasibility of the growing package of specifications and plans is evaluated by independent
experts. Shortfalls in feasibility evidence are treated as risks and addressed by risk
management plans. If the system’s success-critical stakeholders find the risks acceptable and
the risk management plans sound, the project will proceed to the next phase. If not, the
project can extend its current phase, de-scope its objectives, or avoid low-return resource
commitments by terminating the project. If the risks of proceeding straight into development
are negligible, the project can skip one or more early phases.
Emphasis on feasibility evidence deserves further elaboration. A significant step forward in
DoD systems engineering and acquisition guidance was to progress from schedule-based major
project reviews (the Preliminary Design review is scheduled for September 30, so we’ll have it
then, ready or not) to event-based reviews (the PDR will be held when there is a preliminary
design to review). This is better, but frequently leads to “Death by PowerPoint and UML”
reviews. These present much detail, but there is little time to determine whether or not the
design will meet the system’s key performance parameters. Such evidence of feasibility is
generally desired, but not considered as a project deliverable. Thus, it is often neglected when
budgets are tight and contractual progress payments and award fees are based on producing a
design.
In an ICM evidence-based review, the feasibility evidence is a first-class deliverable. As such,
its planning and preparation becomes subject to earned value management and factored into
progress payments and award fees. Investments in feasibility evidence have been found to pay
off significantly in development rework avoidance. In a regression analysis of 161software
projects, rework due to shortfalls in architecture and risk resolution grew from 18% added effort
for small (10,000 source lines of code) projects, to 91% added effort for very large (10 million
SLOC) projects [20].
0.5
Overview of the Generic ICM and DoD Life Cycle Processes
This section presents an overview of the generic ICM life cycle process, with corresponding
DoDI 5000.02 phases and milestones shown along with their generic ICM counterparts. Figure
0-1 reproduces DODI 5000.02’s Figure 1 [8] and Figure 0-2 shows its relation to the generic
ICM phases. As DoDI 5000.02 focuses on acquisition and not operations, the ICM extends
Figure 1 to show reviews for commitments to operations. Section 0.6 will show a more specific
DoD version of the ICM based on Figure 2 of DoDI 5000.02.
The ICM builds on the strengths of current process models: early verification and validation
concepts in the V-model, concurrency concepts in the Concurrent Engineering model, lighterweight concepts in the Agile and Lean models, risk-driven concepts in the spiral model, the
phases and anchor points in the Rational Unified Process (RUP) [9, 10], and recent extensions of
the spiral model to address SoS acquisition [11].
In comparison to the software-intensive RUP (the closest widely-used predecessor to the ICM),
the ICM also addresses hardware and human factors integration. It has extended the RUP phases
to cover the full system life cycle: an Exploration phase focused on Needs and Opportunities (as
in DoDI 5000.02) precedes the RUP Inception phase, which is refocused on valuation and
106753919
-5-
Version Date: 12/31/08
Executive Summary
Version 0.5
investment analysis (and for DoD, on Materiel Solution Analysis and Analysis of Alternatives).
Following a successful DoD Milestone A review based on evidence of system feasibility, the
ICM counterpart of the RUP Elaboration phase and DoD Technology Development and
Capability Development Document phase is focused on Foundations. This is a term based on
[12] describing concurrent development of requirements, architecture, plans, and feasibility
evidence as the essential foundations for the DoD commitment at Milestone B to proceed into
Engineering and Manufacturing Development.
The RUP Construction and Transition phases are combined into Engineering and Manufacturing
Development; and an additional ICM Operations&Production phase combines operations,
production, maintenance, and phase-out. The names of the ICM milestones are changed to
emphasize that their objectives are to ensure stakeholder commitment to proceed to the next level
of resource expenditure based on a thorough feasibility and risk analysis, and not just on the
existence of a set of system objectives and a set of architecture diagrams. Thus, the RUP Life
Cycle Objectives (LCO) milestone and DoD Milestone A is called the Foundations Commitment
Review (FCR) in the ICM, and the RUP Life Cycle Architecture (LCA) milestone and DoD
Milestone B is called the Development Commitment Review (DCR).
Figure 0-1. The Defense Acquisition Management System as in DODI 5000.02
106753919
-6-
Version Date: 12/31/08
Executive Summary
Version 0.5
Figure 0-2. Overview of the Generic Incremental Commitment Life Cycle Process.
(Note: MDD, A, B, and C are the DoDI 5000.02 acquisition milestones.)
106753919
-7-
Version Date: 12/31/08
Executive Summary
Version 0.5
In comparison to the sequential waterfall [13] and V-model [14], the ICM explicitly:

Begins with exploring Needs and Opportunities, as in DoDI 5000.02, rather than with
documenting requirements.

Emphasizes concurrent engineering of requirements and solutions

Establishes Feasibility Evidence Descriptions as project deliverables and milestone
criteria

Enables risk-driven avoidance of unnecessary documents, phases, and reviews

Provides support for a stabilized current-increment development performed
concurrently with a separate change processing and rebaselining activity to prepare
for appropriate and stabilized development of the next increment.
These aspects can be integrated into a waterfall or V-model, enabling projects required to use
such models to cope more effectively with current and future challenges.
The overall lifecycle process divides naturally into two major stages. Stage I, Incremental
Definition, covers the up-front growth in system understanding, definition, feasibility assurance,
and stakeholder commitment leading to a larger Stage II commitment to implement a feasible set
of specifications and plans for Incremental Development and Operations of a materiel solution.
Stage I: The duration of the generic Stage I can be anywhere from one week to five years. The
duration depends on such factors as the number, capability, and compatibility of the proposed
system's components and stakeholders. A small, well-jelled agile-methods, developer-customer
team operating on a mature infrastructure can form and begin incremental development using
Scrum, eXtreme Programming (XP), Crystal or other agile methods in a week. An ultra-large,
unprecedented, multi-mission, multi-owner SoS project may take up to five years to progress
from a system vision through sorting out needs, opportunities, and organizational roles; maturing
key technologies; reconciling infrastructure incompatibilities; reducing uncertainties and risks
via models, simulations, prototypes, advanced technology demonstrations, and operational
exercises; and evolving a feasibility-validated set of specifications and plans for Stage II. These
specifications and plans would be at the build-to level for the initial increment, but only
elaborated into detail for the later increments and the overall system where there were high-risk
elements to resolve and likely changes in mission objectives, technology, and interoperating
systems to accommodate.
As shown in Figure 0-2, each project's activity trajectory will be determined by the arrows
chosen based on the risk assessments and stakeholder commitment decisions at its anchor point
milestone reviews. The small agile project will follow the negligible-risk arrows at the bottom of
Figure 0-2 to skip the Valuation and Foundations phases and begin Stage II after a short
exploratory phase confirms that the risks of doing so are indeed negligible. The ultra-large
project could, for example, apply a form of competitive prototyping to fund four small
competitive concept-definition and validation contracts in the Exploratory phase, three larger
follow-on Valuation contracts, and two considerably larger Foundations contracts, choosing at
each anchor point milestone the best-qualified teams to proceed, based on the feasibility and risk
evaluations performed at each anchor point milestone review. Or, in some cases, the reviews
might indicate that certain essential technologies or infrastructure incompatibilities needed more
work before proceeding into the next phase.
106753919
-8-
Version Date: 12/31/08
Executive Summary
Version 0.5
Stage II: For Stage II, Incremental Development and Operations, a key decision that is made at
the Development Commitment Review is the length of the increments to be used in the system's
development and evolution. A small agile project can use two-to-four week increments.
However, an ultra-large SoS project with a couple of dozen system suppliers, each with a halfdozen levels of subcontractors, would need increments of up to two years to develop and
integrate an increment of operational capability, although with several internal integration subincrements. Some of the non-subsettable hardware systems would take even longer to develop
their initial increments, and would be scheduled to synchronize their deliveries with later
infrastructure or software increments.
The features in each Stage II increment would be prioritized and the increment architected to
enable what has variously been called timeboxing, time-certain development, or Schedule As
Independent Variable (SAIV), in which borderline-priority features are added or dropped to keep
the increment on schedule. It would also be architected to accommodate foreseeable changes,
such as user interfaces or transaction formats. For highly mission-critical systems, it would
include a continuous verification and validation team analyzing, reviewing, and testing the
evolving product to minimize delayed-defect-finding rework.
While the stabilized development team is building the current increment and accommodating
foreseeable changes, a separate system engineering team is dealing with sources of unforeseeable
change and rebaselining the later increments' specifications and plans. Such changes can include
new COTS releases, previous-increment usage feedback, current-increment deferrals to the next
increment, new technology opportunities, or changes in mission priorities. Having the
development team try to accommodate these changes does not work, as it destabilizes their
schedules and carefully-worked-out interface specifications. At the end of each increment, the
system engineering team also produces for expert review the feasibility evidence necessary to
ensure low-risk, stabilized development of the next increment by the build-to-spec team.
0.6
ICM Metaphor and Competitive Prototyping Example
A simple metaphor to help understand the ICM is to compare ICM to incremental-commitment
gambling games such as poker or blackjack, rather than to single-commitment gambling games
such as Roulette. Many system development contracts operate like Roulette, in which a full set
of requirements is specified up front, the full set of resources is committed to an essentially
fixed-price contract, and one waits to see if the bet was a good one or not. With the ICM, one
places an increasing series of bets to see whether the prospects of a win are good or not, and
decides whether or not to continue betting based on better information about the prospects of
success.
An example to illustrate ICM incremental-commitment benefits is the Unmanned Aerial Vehicle
(UAV) (or Remotely Piloted Vehicles (RPV)) system enhancement discussed in Chapter 5 of the
NRC Human-System Integration report [7]. The RPVs are airplanes or helicopters operated
remotely by humans. These systems are designed to keep humans out of harm’s way. However,
current RPV systems are human-intensive, often requiring two people to operate a single vehicle.
The increase in need to operate numerous RPVs is causing a strong desire to modify the 1:2 (one
vehicle controlled by 2 people) ratio to allow for a single operator to operate more than one RPV
If a proof-of principle agent-based prototype demo shows how a single operator can control 4
106753919
-9-
Version Date: 12/31/08
Executive Summary
Version 0.5
RPVs (a 4:1 ratio) in performing some relevant RPV tasks with the help of autonomous agent
technology, how should one proceed?
A total-commitment approach. A total-commitment approach that is representative of several
recent DoD acquisition programs would use the demo results to create a proposed program that
uses the agent-based technology to develop a 4:1 ratio system that enables a single operator to
control 4 RPVs. A number of assumptions are made to sell the program at an optimistic cost of
$1 billion and schedule of 40 months. Enthusiasm is such that the program, budget, and
schedule are established, and an RFP is sent out to prospective bidders.
The winning bidder provided an even more impressive demo of agent technology and a proposal
indicating that all of the problems were well understood, that a preliminary design review (PDR)
could be held in 120 days, and that the cost would be only $800M. However, at the PDR, the
contractor could not show feasible solutions for several key scenarios, such as coping with
inconsistent data, having the individual controllers also coordinate with each other, and
satisfying various interoperability protocols. Since the schedule was tight and the contractor
provided ideas for addressing the problems, the PDR was declared to be passed.
However, the problems were individually more complex than expected and jointly coupled in
critical ways, and after 40 months and $800K, some components were developed but
experiencing integration problems, and even after descoping the performance to a 1:1
operator:RPV ratio, several problems were still unresolved. Eventually, the 1:1 capability was
achieved and the system delivered, but with reduced functionality, a cost of $3 billion, and a
schedule of 80 months.
An incremental-commitment approach. An incremental-commitment approach, using the
ICM process and competitive prototyping, would begin by investing $25M in a pre-MDD
Exploration phase. This would initially involve determining key performance parameters and
scenarios, including inconsistent data, operator collaboration, and interoperability scenarios, and
evaluation criteria and processes. It would then select four bidders to develop virtual prototypes
addressing the key issues and providing evidence of the level of performance. The program
office would then have a set of independent experts evaluate the bidders’ results. Based on the
results, it would determine whether the technology was too immature to merit further current
investment as an acquisition program, or that there were acceptable levels of system
performance, cost, and risk to invest a next level of resources in addressing the problems
identified and developing initial prototype physical capabilities.
In this case, the prospects for developing a 4:1 capability were clearly unrealistic, but the
prospects for a 1:1 capability were sufficiently attractive to merit another level of investment,
corresponding to a Material Solution Analysis - Validation phase. This phase was funded at $75
million, some of the more ambitious key performance parameters were scaled back, and the
competitors were downselected to three. The evaluation of the resulting prototypes served as an
Analysis of Alternatives, and the top two competitors provided sufficient evidence of feasibility
that a Milestone A review was passed, and $225 million was provided for a Technology
Development – Foundations phase.
In this phase, the competitors not only developed operational prototypes and provided evidence
of their ability to satisfy the key performance parameters and scenarios. They also developed
Capability Development Documents defining the foundations for a successful Engineering and
Manufacturing Development (EMD) phase. These included the ICM Development Commitment
106753919
- 10 -
Version Date: 12/31/08
Executive Summary
Version 0.5
Review contents of the proposed system’s concept of operation, requirements, architecture, and
plans, along with a Feasibility Evidence Description providing evidence that a system built to the
architecture would satisfy the requirements and concept of operation, and be buildable within the
budgets and schedules in the plan.
The feasibility evidence included a few shortfalls, such as remaining uncertainties in the
interface protocols with some interoperating systems, but each of these was covered by a risk
mitigation plan in the winning competitor’s submission. The resulting Milestone B review was
passed, and the winner’s proposed $675 million, 18-month, three-increment EMD-phase plan to
develop an Initial Operational Capability was adopted. The resulting IOC was delivered on
budget and schedule, with a few lower-priority features deferred to later system increments.
While this is a hypothetical case, it shows how a premature total commitment without
significant modeling, analysis, and feasibility assessment will often lead to large overruns in
costs and schedule, and a performance that is considerably less than initially desired. However,
by “buying information” early, the incremental commitment and competitive prototyping
approach was able to develop a system with much less late rework than the total commitment
approach, and with much more visibility and control over the process. The competitive
prototyping approach spent about $150-160 million in unused prototypes, but the overall
expenditure was only $1 billion as compared to $3 billion for the total-commitment approach,
and the capability was delivered in 40 versus 80 months, which indicates a strong return on
investment. Further, the funding organizations had realistic expectations of the outcome, so that
a 1:1 capability was a successful realization of an expected outcome, rather than a disappointing
shortfall from a promised 4:1 capability.
A well-documented real example of the approach is the USAF-TRW CCPDS-R project
described in Appendix D of Walker Royce’s book, Software Project Management [21].
0.7
The DoD Version of the ICM
Another view of the DoD version of the ICM used in this Guide is shown in Figure 0-3. It has a
few differences from the corresponding Figure 2 in DoDI 5000.02, provided here for comparison
as Figure 0-4. Each fits certain situations and is a reasonable approximation for the others. For
example, the ICM figure places the beginning of systems engineering preparation for the next
increment of Engineering and Manufacturing Development (EMD) at the beginning of the
current EMD increment, while the DoDI 5000.02 version shows a gap in systems engineering
activity for the next increment.
This will be the case if the content of the next increment requires not-yet mature technology.
However, there will also be situations in which, as recommended in the NRC Pre-Milestone A
and Early-Phase Systems Engineering report [3], there is mature technology available for several
future increments, and evolutionary development is used to reduce the risk of systems
obsolescence in an era of rapid change. In such situations, the major risks are due to effects of
rapid changes in the threat, available technology, mission priorities, interoperating systems, or
previous-increment usage feedback. There, it is important to devote a continuous critical mass
of systems engineering talent to rebaseline the Capabilities Development Document for the next
increment. The ICM’s risk arrows accommodate both situations.
106753919
- 11 -
Version Date: 12/31/08
Executive Summary
Version 0.5
While the scope of DoDI 5000.02 is focused on the operation of the Defense acquisition system,
the ICM scope also includes the pre-acquisition activities leading up to a Materiel Development
Decision (MDD), and the post-acquisition activities involving the system’s operational use.
Thus, the ICM has an Exploration Commitment Review that covers stakeholder commitment to
prepare for an MDD (called an MDP review in Figure 0-2), and an Operations Commitment
Review (OCR) for each increment that is different from the DoDI 5000.02 Milestone C (also
called an ICM Production Commitment Review), which covers stakeholder commitment to
production of the next technology increment. Figure 0-3 shows how the DoD version of the
ICM accommodates these differences. As usual, a single diagram cannot cover all the possible
sequences of operations and production decisions, but these can be covered for a particular
system by separate numbering schemes for operational and production increments.
Figure 0-3. DoD Version of ICM for Evolutionary Deveolpment
106753919
- 12 -
Version Date: 12/31/08
Executive Summary
Version 0.5
Figure 0-4. Requirements and Acquisition Flow as in DoDI 5000.02
0.8
What is Being Concurrently Engineered in the ICM?
Having addressed the stages, phases, and milestones in the generic ICM and its DoD version, let
us now look at the activities. The top row of Activities in Figure 0-2 indicates that a number of
system aspects are being concurrently engineered at an increasing level of understanding,
definition, and development. The most significant of these aspects are shown in Figure 0-5, an
extension of a similar view of concurrently engineered software projects developed as part of the
RUP [9]. As discussed above, the ICM extensions to begin with Envisioning Opportunities and
Understanding Needs precisely correspond to the initial activities discussed in Section 3 of DoDI
5000.02 Enclosure 2 [8].
106753919
- 13 -
Version Date: 12/31/08
Executive Summary
Version 0.5
Figure 0-5. ICM Activity Categories and Level of Effort
As with the RUP version, it should be emphasized that the magnitude and shape of the levels of
effort will be risk-driven and likely to vary from project to project. In particular, they are likely
to have mini risk/opportunity-driven peaks and valleys, rather than the smooth curves shown for
simplicity in Figure 0-5. The main intent of this view is to emphasize the necessary concurrency
of the primary success-critical activities shown as rows in Figure 0-5. Thus, in interpreting the
Exploration column, although system scoping is the primary objective of the Exploration phase,
doing it well involves a considerable amount of activity in understanding needs, envisioning
106753919
- 14 -
Version Date: 12/31/08
Executive Summary
Version 0.5
opportunities, identifying and reconciling stakeholder goals and objectives, architecting
solutions, life cycle planning, evaluating alternatives, and negotiating stakeholder commitments.
For example, if one were exploring the initial scoping of the RPV example discussed above, one
would not just interview a number of stakeholders and compile a list of their expressed mission
needs. One would also envision and explore opportunities for using alternative technologies,
perhaps via competitive prototyping. In the area of understanding needs, one would determine
relevant key performance parameters, scenarios, and evaluation criteria for evaluating the
prototypers’ results. And via the prototypes, one would explore alternative architectural concepts
for developing and evolving the system; evaluate their relative feasibility, benefits, and risks for
stakeholders to review; and if the risks and rewards are acceptable to the stakeholders, to
negotiate commitments of further resources to proceed into a Valuation phase with a clearer
understanding of what level of requirements would be feasible.
0.9
How is All This Concurrent Engineering Synchronized and
Stabilized?
Figure 0-5 indicates that a great deal of concurrent activity occurs within and across the various
ICM phases. To make this concurrency work, the evidence-based anchor point milestone
reviews are the mechanisms by which the many concurrent activities are synchronized,
stabilized, and risk-assessed at the end of each phase. Each of these anchor point milestone
reviews, labeled at the top of Figure 0-5, is focused on developer-produced and independentexpert reviewed evidence, instead of assertions, assumptions, PowerPoint charts and Unified
Modeling Language (UML) diagrams, to help the key stakeholders determine the next level of
commitment.
The review processes and use of independent experts are based on the highly successful AT&T
Architecture Review Board procedures described in [16]. Figure 0-6 shows the content of the
Feasibility Evidence Description. Showing feasibility of the concurrently-developed elements
helps synchronize and stabilize the concurrent activities.
Feasibility Evidence Description Content
Evidence provided by developer and validated by independent experts that if the system is built to
the specified architecture, it will:
–
Satisfy the requirements: capability, interfaces, level of service, and evolution
–
Support the operational concept
–
Be buildable within the budgets and schedules in the plan
–
Generate a viable return on investment
–
Generate satisfactory outcomes for all of the success-critical stakeholders
–
Resolve all major risks by treating shortfalls in evidence as risks and covering
them by risk management plans
–
Serve as basis for stakeholders’ commitment to proceed
Figure 0-6. Feasibility Evidence Description Content
106753919
- 15 -
Version Date: 12/31/08
Executive Summary
Version 0.5
The Operations Commitment Review (OCR) is different, in that it addresses the often much
higher operational risks of fielding an inadequate system. In general, stakeholders will
experience a factor of two-to-ten increase in commitment level in going through the sequence of
ECR/MDP to DCR/B milestones, but the increase in going from DCR/B to OCR can be much
higher, as can be the increase in going from DCR to the Production Commitment Review (PCR
or Milestone C in DoDI 5000.02). These commitment levels are based on typical cost profiles
across the various stages of the acquisition life-cycle. The OCR focuses on evidence of the
adequacy of plans and preparations with respect to doctrine, organization, training, materiel,
leadership, personnel, and facilities (DOTMLPF), along with plans, budgets, and schedules for
fielding and operations.
Taking this to the next level for “directed” System of Systems (SoS) development (the case
where there is an SoS engineering team with considerable responsibility and authority to “direct”
the development of the SoS components), Figure 0-7 shows how these anchor point milestone
reviews can be used to synchronize, stabilize, and manage risks across multiple
supplier/vendor/strategic partner activities.
Figure 0-7. An Example Combining Directed SoS Engineering and Component Supplier
Processes using ICM Anchor Point Reviews.
The major SoS-level milestones are compatible with those of Figure 0-2, but the realities of
many SoSs involve some reinterpretation of their nature. Many SoSs will need to include COTS,
legacy, or separately managed systems that are defined and incrementally released on different
schedules than the SoS in Figure 0-7. The case that is shown in Figure 0-7 is one in which for
reasons of training, provisioning, or operational stability, the main upgrades are batched into
major SoS operational releases. Other SoS cases may require a more continual stream of
upgrades such as security patches or electronic warfare countermeasures, in which parts of the
SoS are more continuously versus incrementally evolving.
Since not all of the source-selected component systems are defined on the same schedule, there
will be delays in reconciling them into a common system architectural framework, requiring an
additional SoS Rebaseline/Adjustment Foundations Commitment Review after the original SoS
FCR/A milestone used to select superior alternatives. The selected suppliers and partners will
106753919
- 16 -
Version Date: 12/31/08
Executive Summary
Version 0.5
also need to participate in negotiating the particular SoS build-to architecture that will be
baselined at the SoS DCR/B milestone. And their incremental delivery schedules will not all be
compatible with the incremental delivery schedule of the SoS in Figure 0-7.
Some of these suppliers, such as Supplier x, will already be operating and will feed incremental
upgrade information into the SoS process during its definition and development stages. The SoS
system engineers will try to predefine and anticipate these upgrades as much as possible, but will
have to adapt to changes in Supplier x's capabilities or interfaces. If these are well-anticipated,
the SoS development team in Figure 0-2 will accommodate them. If they are complex and
unanticipated, threatening destabilization of the build-to-spec team, the SoS architecting
rebaselining team will engineer an interim solution if necessary to keep the operational capability
running. If a more comprehensive solution is needed for the longer term, the team will
rebaseline the SoS architecture of the next increment to accommodate Supplier x's new
capability. Changes from Strategic Partner C and Supplier B would be handled similarly.
In some additional cases, particularly for Supplier A who may be developing a longer-duration,
non-subsettable, hardware-intensive initial operational capability, the SoS management will
schedule Supplier A's IOC to synchronize with a later SoS increment (OCR2 in Figure 0-7).
Even more complex are the “Acknowledged” and “Collaborative” SoS forms, which contain
multiple system owners and management chains to deal with. Chapter 14 discusses how the
ICM can be tailored to address these increasingly important SoS forms.
0.10
Project Experience with ICM Principles
One of the origins of the ICM was an effort during its formulation to capture commercial best
practices for incorporating human-system factors into the system life cycle process in the NRC
study on that topic [7]. A good example of commercial use of these best practices in an ICM
process framework is an award-winning, commercially-successful medical infusion pump
development involving complex integration of leading-edge hardware, software, and human
factors technology for a safety-critical product. It is documented in Chapter 5 of [7].
Another good source of successful projects that have applied ICM Critical Success Factor
principles is the annual series of Top-5 software-intensive systems projects published in
CrossTalk [17].
The “Top-5 Quality Software Projects” were
chosen annually by panels of leading experts as
role models of best practices and successful
Year
Concurrent
RiskEvolutionary
Engineering Driven
Growth
outcomes. Table 0-1 summarizes each year’s
record with respect to usage of four of the six
2002
4
3
3
principles: concurrent engineering; risk-driven
2003
5
4
3
activities; and evolutionary, iterative system
2004
3
3
4
growth (most of the projects were not specific
about system sponsor commitment and
2005
4
4
5
accountability or stakeholder satisficing). Of
Total (of 20)
16
14
15
the 20 Top-5 projects in 2002 through 2005, 16
explicitly used concurrent engineering; 14
explicitly used risk-driven development; and 15 explicitly used evolutionary, iterative system
Table 0-1. Number of Top-5 Projects
Explicitly Using ICM Principles
106753919
- 17 -
Version Date: 12/31/08
Executive Summary
Version 0.5
growth, while additional projects gave indications of their partial use. Evidence of successful
results of stakeholder-satisficing can be found in the annual series of University of Southern
California (USC) e-Services projects using the Win-Win Spiral model as described in [18].
Since 1998, over 50 user-intensive e-Services applications have used this precursor and
subsequent evolving versions of the ICM to achieve a 92% success rate of on-time delivery of
stakeholder-satisfactory systems.
0.11
Structure and Content of the Guide
The Guide comprises of 21 chapters, not including this Executive Summary. Below is a short
overview of each chapter.
Section I. ICM Fundamentals
Chapter 1: ICM Fundamentals – Fundamental Principles. This chapter deals specifically
with the six key principles at the core of the ICM: 1 Commitment and accountability of system
engineers, developers, and managers; 2. Stakeholder satisficing; 3. Incremental and
evolutionary growth of system definition and stakeholder commitment; 4. Iterative system
development and definition; 5. Concurrent system definition and development; and 6. Evidencebased, risk driven decision milestones. It includes more thorough explanations of these
principles in the context of the ICM, along with examples of how these were used successfully in
some of the CrossTalk “Top-5 Quality Software Projects.”
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs. The DoDI 5000.02
counterparts of the ICM AP (Anchor Point) Milestones and FEDs (Feasibility Evidence
Descriptions) are discussed in this chapter. They are the Materiel Decision
Preparation/Exploration Commitment Review (MDP/ECR), the Materiel Development
Decision/Valuation Commitment Review (MDD/VCR), the Milestone A/Foundations
Commitment Review (A/FCR), the Development Commitment Review (Milestone B/DCR), the
DoDI 5000.02 Milestone C Production and Deployment commitment review (C/PCR) and the
ICM Operations Commitment Review (OCR). This chapter will explain the milestones and
FEDs on a fundamental level, e.g. What are examples of FEDs? Why is an FED required at each
Anchor Point? How do they help synchronize and stabilize concurrent activities? How do they
help in risk management? How is their progress tracked as a basis of earned value through each
phase?
Chapter 3: ICM Fundamentals – Incremental Adoptability. Current system development
programs are already operating under a set of contractual standards that may not fit well with
their need to operate as or within a system of systems; to handle emergent requirements; or to
cope with rapid change. For such programs, it is often not possible to make major process
changes all at once. Rather, it is better to incrementally change processes over time. This
chapter provides guidance for incremental adoption of key ICM processes.
106753919
- 18 -
Version Date: 12/31/08
Executive Summary
Version 0.5
Section II. ICM Stage I: Incremental Definition
Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration
Phase. The ICM Exploration Phase is focused on system scoping based upon the need for new
or upgraded capabilities. This phase explores materiel development and other alternatives by
investigating and understanding needs, envisioning opportunities, identifying and reconciling
stakeholder goals and objectives, negotiation of stakeholder commitments, and preparing for a
Materiel Development Decision by such activities as establishing guidelines for the analysis of
alternatives. This chapter describes these activities in more detail.
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table. As with
other models trying to address a wide variety of situations, the general form of the ICM is rather
complex. However, its risk-driven nature has enabled us to determine a set of twelve common
risk patterns and organize them into a decision table that can help projects new to the ICM
converge on a process that fits well with their particular process drivers. For each of the twelve
special cases, the decision table provides top-level guidelines for tailoring the key activities of
the ICM, along with suggested lengths between each internal system build and each external
system increment delivery. This chapter elaborates on each of the twelve cases and provides
examples of their use.
Chapter 6: Stage I: Incremental Definition – The Materiel Solution Analysis &
AoA/Valuation Phase. If the results of the MDD Preparation/Exploration Phase indicate that
there are system capability needs worth pursing further, the Materiel Development Decision
(MDD) is passed positively, and a commitment is made to go on to the next phase. The Materiel
Solution Analysis & AoA/Valuation Phase is focused on further system definition, valuation and
investment analysis. As part of this effort, an analysis of alternatives (AoA) is conducted using
the guidance provided in the MDD Preparation/Exploration Phase. This chapter describes these
activities and illustrates them though examples in more detail.
Chapter 7: Stage I: Incremental Definition – The Technology Development &
CDD/Foundations Phase. If the results of the ICM Materiel Solution Analysis & AoA
Valuation phase indicate that there are viable, cost-effective options for providing the needed
capability, a commitment is made to move forward into the Technology Development &
CDD/Foundations Phase. For the DoD acquisition process, this Commitment is the same as the
Milestone A commitment. The TD&CDD/Foundations Phase focuses not only on technology
development but also on the concurrent development of a Capability Development Document
(CDD) covering the requirements, architecture, and plans needed to provide a solution. These
are the needed “foundations” upon which the system will be developed in Stage II, and whose
feasibility evidence will be assessed in making a Milestone B/ Development Commitment
Review decision to proceed into Engineering and Manufacturing Development. This chapter
describes these activities and illustrates them through examples in more detail.
106753919
- 19 -
Version Date: 12/31/08
Executive Summary
Version 0.5
Section III. ICM Stage II: Incremental Development
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V. If a
viable, cost effective solution is identified in the TD&CDD/Foundations Phase, a commitment is
made to proceed with development. For the DoD acquisition process, this commitment is the
same as the Milestone B commitment. The Engineering and Manufacturing Development Phase
focuses on the development of the system (or deployable increment of a system) employing
continual V&V activities to ensure a successful development. This chapter describes and
illustrates these activities in more detail.
Chapter 9: Stage II: Incremental Development – Concurrent Next-Increment Rebaseline.
A key feature of the ICM is that as the current increment is undergoing stable development, a
parallel team is handling unanticipated change traffic and rebaselining the foundations for the
next increment of development. Note that under the DoDI 5000.02 acquisition process, these
activities will generally require additional Milestone A and/or Milestone B Commitments,
depending upon the scope of the next planned increment and the maturity of the planned
technologies for the next increment. This chapter describes and illustrates these activities in more
detail.
Section IV. Key ICM Practices and Applications
Chapter 10: Key ICM Practices and Applications – Competitive Prototyping. This chapter
discusses how the ICM supports competitive prototyping (CP) in the DoD context. It also
provides guidance for assessing the costs associated with CP as “buying information to reduce
risk”, determining “how much CP is enough?” and for dealing with multiple rounds of CP, as in
the RPV CP example above.
Chapter 11: Key ICM Practices and Applications – Stakeholder Satisficing. This chapter
describes the role of stakeholder satisficing in the successful definition and development of
system capabilities for large systems and systems of systems with multiple stakeholders. It
provides guidance on identifying success-critical stakeholders, and addressing the most likely
conflicts among the various classes of key DoD stakeholders. Examples are provided that show
how various trades and negotiations can be conducted.
Chapter 12: Key ICM Practices and Applications – Risk Assessment and Management.
Realistic risk assessment at each anchor point and active risk management during each phase are
key to deciding when to proceed to the next phase, when to investigate alternatives further before
proceeding to the next phase, and when to discontinue because the alternatives are too risky.
This chapter provides guidance and examples of the role of risk assessment and management at
each of the Commitment review points, for determining how much of an activity or artifact is
enough, and for such practices as CP.
Chapter 13: Key ICM Practices and Applications – Estimation and Management Metrics.
As processes evolve to handle current and future challenges, so too must estimation techniques
and management metrics. Some of these challenges include emergent requirements, rapid
change, net-centric system of systems, model-driven system development, service oriented
106753919
- 20 -
Version Date: 12/31/08
Executive Summary
Version 0.5
architectures, Brownfield development, always-on never-fail system development, competitive
prototyping, hardware/software/systems engineering integration, complex systems, development
process changes that impact estimation processes, and shifts in key work products used to
communicate information between various system development participants. This chapter
explores options and provides best-available guidance for meeting challenges such as these.
Chapter 14: Key ICM Practices and Applications – ICM for Systems of Systems. Recent
DoD research [22] has found that traditional processes need to be adapted to handle the special
challenges presented by the creation and evolution of systems of systems. This research has also
identified several types of SoS, with each type characterized by the level of authority and
responsibility used to guide the evolution of the SoS. ICM’s flexible and tailorable approach to
system development processes is well-suited for these various types of SoSs. This chapter
examines various process approaches and examples for the “acknowledged” or “directed” SoS
types identified in [22], as well as general support guidance for the “collaborative” SoS type.
Chapter 15: Key ICM Practices and Applications – Implications for Acquisition
Management and Staffing. The ICM’s emphasis in Stage I on concurrent engineering and on
evidence-based versus schedule-based or event-based reviews has several key implications for
how to manage and staff the pre-Milestone B activities. These include higher levels of
investment but often shorter schedules for systems engineering of complex projects; new
management metrics to track as discussed in Chapter 13; and ways of funding Stage I that ensure
continuity of effort but also the ability to redirect or discontinue efforts with too-high or
unacceptable risk of continuing on their current course. The ICM’s emphasis in Stage II is on
concurrent stable development of the current increment; continuous V&V of the current
increment; and a parallel team handling unanticipated change traffic and rebaselining the
foundations for the next increment of development. This implies the need for different
contractual vehicles, incentive structures, and staffing career paths for the three concurrent
classes of activity.
Section V. Key ICM System Definition Artifacts
Chapters 16 through 21: Key System Definition Artifacts. These chapters provide guidance
and examples for the key ICM system definition artifacts. These include the operational concept
definition, requirements description, architecture description, life cycle plan, systems
engineering management plan, and feasibility evidence descriptions at different points in the life
cycle.
0.12
Conclusions
Future mission-critical DoD combat platforms, systems of systems, and network-centric services
will have many usage uncertainties and emergent characteristics. Their hardware, software, and
human factors will need to be concurrently engineered, risk-managed, and evolutionarily
developed to converge on cost-effective system operations and mission success.
The recent revision of DoDI 5000.02 provides a much improved policy for dealing with these
future challenges, including its starting with Needs and Opportunities, its emphasis on early
106753919
- 21 -
Version Date: 12/31/08
Executive Summary
Version 0.5
Analysis of Alternatives as an entry condition for Milestone A, its inclusion of a passed
Preliminary Design Review as an entry condition for Milestone B, and its emphasis on
evolutionary versus single-increment or prespecified increments of development.
These emphases on DoDI 5000.02 are also key emphases in the Incremental Commitment Model
(ICM), along with more detailed guidance on how to apply them. The ICM has been developed
based on best commercial practices for addressing such future challenges; on discussions with
contributors to DoDI 5000.02; on successful use of the ICM principles on previous DoD
projects; and on its exploratory use on highly complex, future-precursor, net-centric DoD
systems of systems.
The ICM builds on the experience-based critical success factor principles of stakeholder
satisficing, incremental definition, iterative evolutionary system growth, concurrent engineering,
and evidence-based, risk-driven management milestones. It is not a one-size-fits-all process
model, but has risk-driven options at each milestone decision point to enable projects to adapt to
their particular situations. It provides a decision table for early determination of common specialcase life cycle processes to avoid overkill in project ceremony and documentation.
It also incorporates the strengths of existing V, concurrent engineering, spiral, agile, and lean
process models. As with these other models, it is not fully tested for its ability to support DoDI
5000.02 in all project situations, but it has been formulated to make it straightforward to do so.
0.13
References
1. Krygiel, A. (1999) Behind the Wizard’s Curtain; CCRP Publication Series, July.
2. Young, J. (2007) Prototyping and Competition, USD(AT&L) Memorandum, 19
September.
3. Kaminski, P. et al.(2008) Pre-Milestone A and Early-Phase Systems Engineering, The
National Academies Press.
4. Boehm, B. (2000) “Unifying Software Engineering and Systems Engineering”,
Computer, March 2000, pp. 114-116.
5. Clements, P. et al. (2002). Evaluating Software Architectures, Addison Wesley.
6. Baldwin, K. (2007) "DoD Software Engineering and System Assurance: New
Organization, New Vision,"
http://csse.usc.edu/events/2007/ARR/presentations/Baldwin.ppt, February 14.
7. Pew, R. W., and Mavor, A. S., (2007). Human-System Integration in the System
Development Process: A New Look, National Academy Press.
8. USD(AT&L) (2008) Operation of the Defense Acquisition System, DoD Instruction
5000.02, December 8.
9. Kruchten, P. (1999). The Rational Unified Process, Addison Wesley.
10. Boehm, B. (July 1996). Anchoring the Software Process. Software. pp. 73-82.
11. Boehm, B., and Lane, J. (May 2006). 21st Century Processes for Acquiring 21st Century
Systems of Systems. CrossTalk. pp. 4-9.
106753919
- 22 -
Version Date: 12/31/08
Executive Summary
Version 0.5
12. Rechtin, E. (1991). Systems Architecting, Prentice Hall.
13. Royce, W. W. (1970). Managing the development of large software systems: concepts
and techniques, Proceedings, Wescon 1970.
14. Patterson, F. (1999). System engineering life cycles: life cycles for research,
development, test, and evaluation; acquisition; and planning and marketing, in Sage, A.,
and Rouse, W., ed., (1999) Handbook of Systems Engineering and Management, Wiley,
pp. 59-111.
15. U.S Department of Defense (2003). DoD Instruction 5000.2, Operation of the Defense
Acquisition System.
16. Maranzano, J. et al. (2005) Architecture reviews: Practice and experience. IEEE
Software, March/April 2005, pp. 34-43.
17. CrossTalk (2002-2005), “Top Five Quality Software Projects,” January 2002, July 2003,
July 2004, September 2005.
18. Boehm, B., Egyed, A., Kwan, J., Port, D., Shah, A., and Madachy, R. (July 1998). Using
the WinWin Spiral Model: A Case Study. Computer. pp.33-44.
19. Hopkins, R., and Jenkins, K. (2008) Eating the IT Elephant: Moving from Greenfield
Development to Brownfield, IBM Press.
20. Boehm, B., Valerdi, R., and Honour, E. (2008) The ROI of Systems Engineering: Some
Quantitative Results for Software-Intensive Systems, Systems Engineering, Vol. 11,
No.3, pp. 221-234.
21. Royce, W. E. (1998) Software Project Management. Addison Wesley.
22. OUSD(AT&L)/Systems and Software Engineering (2008), Systems Engineering Guide
for System of Systems, Version 1.0, June.
106753919
- 23 -
Version Date: 12/31/08
Chapter 1: ICM Fundamentals – Fundamental Principles
Version 0.5
1Chapter 1: ICM Fundamentals –
Fundamental Principles
1.1
Introduction
The DoD-sponsored National Research Council (NRC) study on Human–System Integration in
the System Development Process analyzed the strengths and difficulties of current process
models in addressing six challenges: complex, multi-owner, net-centric SoS; emergent
requirements; rapid change; reused components and legacy system constraints; high assurance of
qualities; and increasing importance of software and human factors. Each had strengths, but
needed further refinements to address all of the six challenges. The most important conclusion,
though, was that there were key process principles that address the challenges, and that forms of
the models were less important than their ability to adopt the principles.
1.2
Key Principle #1: Commitment and Accountability of System
Engineers, Developers, and Managers
Without key personnel commitment and accountability for the system under development, there
is no way to build trust among the system’s stakeholders. It is too easy to overpromise and
depart. As recommended in the NRC Kaminski report [1], development increments must be
short enough that the key program leaders remain engaged in their current jobs. And there must
be clear visibility of progress versus plans up and down the supplier chain.
[Further elaboration TBD]
1.3
Key Principle #2: Stakeholder Satisficing
If a system development process presents a success-critical operational or development
stakeholder with the prospect of an unsatisfactory outcome, the stakeholder will generally refuse
to cooperate, resulting in an unsuccessful system. Stakeholder satisficing includes identifying
the success-critical stakeholders and their value propositions; negotiating a mutually satisfactory
set of system requirements, solutions, and plans; and managing proposed changes to preserve a
mutually satisfactory outcome.
[Further elaboration TBD]
1.4
Key Principle #3: Incremental and Evolutionary Growth of System
Definition and Stakeholder Commitment
This characteristic captures the often incremental discovery of emergent requirements and
solutions via methods such as prototyping, operational exercises, and use of early system
capabilities. Requirements and commitment cannot be monolithic or fully pre-specifiable for
complex, human-intensive systems; increasingly detailed understanding, trust, definition and
commitment is achieved through an evolutionary process.
[Further elaboration TBD]
106753919
- 24 -
Version Date: 12/31/08
Chapter 1: ICM Fundamentals – Fundamental Principles
1.5
Version 0.5
Key Principle #4: Iterative System Development and Definition
The incremental and evolutionary approaches lead to cyclic refinements of requirements,
solutions, and development plans. Such iteration helps projects to learn early and efficiently
about operational and quality requirements and priorities.
[Further elaboration TBD]
1.6
Key Principle #5: Concurrent System Definition and Development
Initially, this includes concurrent engineering of requirements and solutions, and integrated
product and process definition. In later increments, change-driven rework and rebaselining of
next-increment requirements, solutions, and plans occurs concurrently with stabilized
development of the current system increment. This allows early fielding of core capabilities,
continual adaptation to change, and timely growth of complex systems without waiting for every
requirement and subsystem to be defined.
[Further elaboration TBD]
1.7
Key Principle #6: Evidence-Based, Risk Driven Decision Milestones
The key to synchronizing and stabilizing all of this concurrent activity is a set of evidence-based,
risk-driven stakeholder decision milestones. At these milestones, evidence of the business,
technical, and operational feasibility of the growing package of specifications and plans is
evaluated by independent experts. Shortfalls in feasibility evidence are treated as risks and
addressed by risk management plans. If the system’s success-critical stakeholders find the risks
acceptable and the risk management plans sound, the project will proceed to the next phase. If
not, the project can extend its current phase, de-scope its objectives, or avoid low-return resource
commitments by terminating the project. If the risks of proceeding straight into development are
negligible, the project can skip one or more early phases.
[Further elaboration TBD]
1.8
References
1. Kaminski, P. et al.(2008) Pre-Milestone A and Early-Phase Systems Engineering, The
National Academies Press.
106753919
- 25 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
2Chapter 2: DoD-ICM Fundamentals – AP
Milestones & FEDs
2.1
Developing and Using Feasibility Evidence for Life Cycle
Commitment Milestones
Success-Critical Stakeholders (SCSHs) for systems under development have the authority and
the responsibility to exercise accountable governance of their organization’s resources when
committing such resources to the next phase of system development. In many cases, when using
processes other than the ICM, these stakeholders are presented with insufficient information
upon which to make a responsible decision.
As a result, many projects proceed into system development with little idea of whether their
project requirements, architectures, plans, budgets, and schedules are feasible and consistent or
not. This is one of the major reasons for the relatively low (20-30%) success rates for successful
on-time, in-budget system deliveries in the commercial Standish Reports and government GAO
reports.
The ICM provides guidance for avoiding such system overruns and shortfalls by using APs and
FEDs. The main distinction of using APs and FEDs is to provide evidence of feasibility as part
of the system definition’s core deliverables, not just assertions, assumptions, or aspirations
placed in optional appendices.
APs and FEDs emphasize that independent experts should evaluate the evidence supplied by
developers. Any shortfalls in evidence should be identified as sources of project risk. These
sources of project risk need to be covered by risk mitigation plans.
APs and FEDs are an integral part of the ICM discussed in [Boehm-Lane, 2007; Boehm-Lane,
2008; and Pew-Mavor, 2007]. However, they are applicable to models other than the ICM. A
good approximation to their use may be obtained by adding FEDs to the content of traditional
waterfall or V-model milestones such as System Requirements Reviews (SRRs), System
Functional Reviews (SFRs), and Preliminary Design Reviews (PDRs).
2.2
Nature of FEDs and AP Milestones
FEDs are evidence provided by the developer and validated by independent experts that, if the
system is built to the specified architecture it will:
1. Satisfy the specified operational concept and requirements, including capability, interfaces,
level of service, and evolution
2. Be buildable within the budgets and schedules in the plan
3. Generate a viable return on investment
4. Generate satisfactory outcomes for all of the SCSHs
5. Identify shortfalls in evidence as risks, and cover them with risk mitigation plans
106753919
- 26 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
An FED does not assess a single sequentially developed system definition element, but the
consistency, compatibility, and feasibility of several concurrently-engineered elements. To make
this concurrency work, a set of AP milestone reviews are performed to ensure that the many
concurrent activities are synchronized, stabilized, and risk-assessed at the end of each phase.
Each of these AP reviews is focused on developer-produced and expert-validated evidence,
documented in the FED, to help the key stakeholders—the SCSHs—determine whether to
proceed into the next level of commitment. Hence, they are called Commitment Reviews.
The FED is based on evidence from simulations, models, or experiments with planned
technologies and increasingly detailed analysis of development approaches and project
productivity rates. The parameters used in the analyses should be based on measured component
performance or on historical data showing relevant past performance, cost estimation accuracy,
and actual developer productivity rates.
A shortfall in feasibility evidence indicates a level of program execution uncertainty and a source
of program risk. It is often not possible to fully resolve all risks at a given point in the
development cycle, but known, unresolved risks need to be identified and covered by risk
management plans, including the necessary staffing and funding to address them. The nature of
the evidence shortfalls, the strength and affordability of the risk management plans, and the
stakeholders’ degrees of risk acceptance or avoidance will determine their willingness to commit
the necessary resources to proceed. A program with risks is not necessarily bad, particularly if it
has strong risk management plans. A program with no risks may be high on achievability, but
low on ability to produce a timely competitive advantage.
A program more familiar with a sequential waterfall or V-model can achieve most of the effect
of a FED-based anchor point milestone review by adding an FED to the set of artifacts to be
developed and reviewed for adequacy and satisfaction at a System Requirements Review (SRR),
System Functional Review (SFR), or Preliminary Design Review (PDR). In principle, some
guidance documents indicate that such feasibility evidence should be produced and reviewed.
But since the evidence is not generally specified as a developer deliverable and is vaguely
defined, it is generally inadequate as a basis for stakeholder commitments.
2.3
Problems Encountered without FED
In the early 1980s, a large government organization contracted with TRW to develop an
ambitious information query and analysis system. The system would provide more than 1,000
users, spread across a large building complex, with powerful query and analysis capabilities for a
large and dynamic database.
TRW and the customer specified the system using a classic sequential-engineering waterfall
development model. Based largely on user need surveys, an oversimplified high-level
performance analysis, and a short deadline for getting the To-Be-Determineds out of the
requirements specification, they fixed into the contract a requirement for a system response time
of less than one second.
Subsequently, the software architects found that subsecond performance could only be provided
via a highly customized design that attempted to anticipate query patterns and cache copies of
data so that each user’s likely data would be within one second’s reach (a 1980’s precursor of
Google). The resulting hardware architecture had more than 25 super-midicomputers (an earlier
106753919
- 27 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
term for a computer with performance and capacity between a minicomputer and a mainframe)
busy caching data according to algorithms whose actual performance defied easy analysis. The
scope and complexity of the hardware-software architecture brought the estimated cost of the
system to nearly $100 million, driven primarily by the requirement for a one-second response
time.
Faced with this unattractive prospect (far more than the customer’s budget for the system), the
customer and developer decided to develop a prototype of the system’s user interface and
representative capabilities to test. The results showed that a four-second response time would
satisfy users 90 percent of the time. A four-second response time, with special handling for
high-priority transactions, dropped development costs closer to $30 million. Thus, the premature
specification of a 1-second response time neglected the risk of creating an overexpensive and
time-consuming system development. Fortunately, in this case, the only loss was the wasted
effort on the expensive-system architecture and a 15-month delay in delivery (see Figure 2-1).
More frequently, such rework is done only after the expensive full system is delivered and found
still too slow and too expensive to operate.
Figure 2-1. Problems Encountered without FED: 15-Month Architecture Rework Delay
2.4
Problems Avoidable with FED
Had the developers been required to deliver a FED showing evidence of feasibility of the onesecond response time, they would have run benchmarks on the best available commercial query
systems, using representative user workloads, and would have found that the best that they could
do was about a 2.5-second response time, even with some preprocessing to reduce query latency.
They would have performed a top-level architecture analysis of custom solutions, and concluded
106753919
- 28 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
that such 1-second solutions were in the $100 million cost range. They would have shared these
results with the customer in advance of any key reviews, and found that the customer would
prefer to explore the feasibility of a system with a commercially-supportable response time.
They would have done user interface prototyping and found much earlier that the four-second
response time was acceptable 90% of the time.
As some uncertainties still existed about the ability to address the remaining 10% of the queries,
the customer and developer would have agreed to avoid repeating the risky specification of a
fixed response time requirement. The customer and developer would instead define a range of
desirable-to-acceptable response times, with an award fee provided for faster performance.
They would also have agreed to reschedule the next milestone review to give the developer time
and budget to present evidence of the most feasible solution available, using the savings over the
prospect of a $100 million system development as rationale. This would have put the project on
a more solid success track over a year before the actual project discovered and rebaselined itself,
and without the significant expense that went into the unaffordable architecture definition.
For example, the customer and developer would negotiate response time ranges. They would say
two-second response time desirable, four-second response time acceptable with some twosecond special cases. Next they would benchmark commercial system add-ons to validate their
feasibility. Finally, they would present solution and feasibility evidence at AP milestone review.
The result would be an acceptable solution with minimal delay.
2.5
AT&T Experience with AP Reviews
AT&T and its spinoffs (Telcordia, Lucent, Avaya, regional Bell companies) have been
successfully using versions of AP reviews since 1988, see Figure 2-2. On average, there has
been 10% savings per reviewed project, with substantially larger savings on a few reviewed
projects. More detail on their Architecture Review experience is in J. Maranzano et. al.,
“Architecture Reviews: Practice and Experience,” IEEE Software, March/April 2005.
106753919
- 29 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
Figure 2-2. AT&T Experience with AP Reviews
USC has used APs and FEDs to achieve a 92% on-time, in-budget, satisfied-client success rate
on over 50 fixed-schedule campus e-service projects. AP Reviews are also an integral part of the
Rational Software Process, with many successful implementations.
2.6
Overview of ICM Phases, Anchor Points, and Decision Options
Figure 0-1 shows how the ICM spans the life cycle process from concept exploration to
operations. Each phase includes a number of concurrent activities that are synchronized and
stabilized with a FED-based anchor point milestone review at points corresponding with the
major DoD acquisition milestones. At each anchor point, there are 4 options, based on the
assessed risk of the proposed system. Some options involve go-backs. These options result in
many possible process paths.
The life cycle is divided into two stages: Stage I of the ICM (Definition) has 3 decision nodes
with 4 options/node, culminating with incremental development in Stage II (Development and
Operations). Stage II has an additional 2 decision nodes, again with 4 options/node.
One can use ICM risk patterns to generate frequently-used processes with confidence that they fit
the situation. Initial risk patterns can generally be determined in the Exploration phase. One
then proceeds with development as a proposed plan with risk-based evidence at the
VCR/Material Development Decision milestone, adjusting in later phases as necessary.
Risks associated with the system drive the life cycle process. Information about the risks
(feasibility assessments) supports the decision to proceed, adjust scope or priorities, or cancel the
program. The downward arrows at the end of each phase provide built-in off-ramps for clearly
106753919
- 30 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
infeasible programs. The horizontal arrows provide additional options to skip phases or
activities if the risk of doing so is negligible.
2.7
Key Point: Need to Show Evidence
It was a significant step forward for DoD to progress from having schedule-based reviews, where
the review took place at the scheduled time whether the project was ready or not, to event-based
reviews, where the review would not take place until its artifacts—e.g., functional
requirements—were completed.
However, many event-based reviews focused on hastily-produced artifacts that had not been
validated for compatibility or feasibility, such as the example in Figure 2-1. Thus, it is another
significant step forward to progress from event-based reviews to evidence-based reviews.
The major difference in the example project proceeding without and with an FED was that the
need to provide evidence of feasibility requires the project to develop and exercise benchmarks
and prototypes that expose both infeasible solutions and unnecessarily ambitious requirements.
FED needs to be more than just traceability matrices and PowerPoint charts. Evidence can
include results of:

Prototypes: of networks, robots, user interfaces, COTS interoperability

Benchmarks: for performance, scalability, accuracy

Exercises: for mission performance, interoperability, security

Models: for cost, schedule, performance, reliability; tradeoffs

Simulations: for mission scalability, performance, reliability

Early working versions: of infrastructure, data fusion, legacy compatibility

Previous experience

Combinations of the above
Not only does the evidence need to be produced, but it needs to be validated by independent
experts. These experts need to determine the realism of assumptions, the reperesentativeness of
scenarios, the thoroughness of analysis, and the coverage of key off-nominal conditions. The
risk of validating just nominal-case scenarios is shown in the next section.
2.8
Off-Nominal Architecture-Breakers
Often, projects with tight schedules will overfocus their architecture and development effort on
the nominal-case requirements. Figure 2-3 shows the frequent result: a critical off-nominal
condition that cannot be handled by the nominal-case architecture, causing large amounts of
rework and delays, and reproducing the common Pareto diagram in which 20% of the defects
cause 80% of the work. This also happens on programs that suboptimize their architecture to
succeed on the initial increments.
106753919
- 31 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
Figure 2-3. Critical off-Nominal Condition that Cannot Be Handled by the Nominal-Case
Architecture
This analysis enabled TRW not only to extend its risk management guidelines to cover critical
off-nominal scenarios, but also to help convince future customers to reinterpret traditional
waterfall milestones to include validation of feasibility evidence. A resulting example success
story, the CCPDS-R project, is summarized in [Royce, 1998].
2.9
Common Examples of Inadequate Evidence
Common examples of inadequate evidence, or what could be referred to as non-evidence include
these statements:
1. “Our engineers are tremendously creative. They will find a solution for this.”
2. “We have three algorithms that met the Key Performance Paramaters (KPPs) on small-scale
nominal cases. At least one will scale up and handle the off-nominal cases.”
3. “We’ll build it and then tune it to satisfy the KPPs.”
4. “The COTS vendor assures us that they will have a security-certified version by the time we
need to deliver.”
5. “We have demonstrated solutions for each piece from our NASA, Navy, and Air Force
programs. It’s a simple matter of integration to put them together.”
Numerous projects have repeated the consequences of proceeding without evidence that their
plans and specifications are compatible and feasible. Why does this happen? Most often
106753919
- 32 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
because the project has been pushed into an inadequate budget and schedule, particularly for
systems engineering, and because its progress payments and award fees are initially based on the
existence of specifications rather than on their feasibility. As a result, one often hears statements
such as the above in place of evidence at milestone reviews.
What can be done about this? First, enough budget and schedule needs to be provided (models
such as COSYSMO [Valerdi, 2005] are a good cross-check). Next, plans need to be made and
progress monitored for both producing and reviewing the feasibility evidence. And if progress
reviews uncover statements such as the above, corrective action such as the examples in the next
section needs to be performed.
2.10
Examples of Making the Evidence Adequate
These candidate actions in italics can resolve the issues of the previous section:
1. “Our engineers are tremendously creative. They will find a solution for this.”
Have the creative engineers prototype and evaluate a solution on
some key nominal and off-nominal scenarios.
2. “We have three algorithms that met the Key Performance Paramaters (KPPs) on small-scale
nominal cases. At least one will scale up and handle the off-nominal cases.”
Prototype and evaluate the three examples on some key nominal
and off-nominal scenarios.
3. “We’ll build it and then tune it to satisfy the KPPs.”
Develop prototypes and/or simulations and exercise them to show
that the architecture will not break while scaling up or handling
off-nominal cases.
4. “The COTS vendor assures us that they will have a security-certified version by the time we
need to deliver.”
Conduct a scaled-down security evaluation of the current COTS
product. Determine this and other vendors’ track records for
getting certified in the available time. Investigate alternative
solutions.
5. “We have demonstrated solutions for each piece from our NASA, Navy, and Air Force
programs. It’s a simple matter of integration to put them together.”
Have a tiger team prototype and evaluate the results of the simple
matter of integration.
2.11
ICM Detailed Activity Descriptions
Stage II of the Incremental Commitment Life Cycle provides a framework for concurrent
engineering and development of multiple increments.
Note: The term “concurrent engineering” fell into disfavor when behind-schedule developers
applied it to the practice of proceeding into development while the designers worked on finishing
106753919
- 33 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
the design. Not surprisingly, the developers encountered a high rework penalty for going into
development with weak architecture and risk resolution.
“Concurrent engineering” as applied in the ICM is much different. It is focused on doing a costeffective job of architecture and risk resolution in Stage I; and on performing stabilized
development, verification, and validation of the current system increment while concurrently
handling the systems change traffic and preparing a feasibility-validated architecture and set of
plans for the next increment in Stage II.
Figure 2-4 provides more detailed descriptions of the activities concurrently going on during the
various ICM phases. A Development Commitment Review / Milestone B (DCR/MS-B) package
contains all of the Foundations material (in risk-driven levels of detail) needed for its
independent expert review preceding its DCR/MS-B: operations concept, requirements,
architecture, plans, business case, feasibility evidence, and risk management plans as necessary.
Even further detail on the contents of this material is provided next.
106753919
- 34 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
Figure 2-4. ICM Detailed Activity Descriptions
106753919
- 35 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
2.12
Version 0.5
Focus of Each Commitment Review
Commitment Reviews are so called because it reviews the evidence, created in the previous
phase, and decides how to commit: proceed ahead, do more work, or discontinue the program.
Evidence generally comes in the form of review packages, such as the Development
Commitment Package (DC Package), created during the Foundations phase, used during the
OCD, determining whether to proceed to the Development phase. These packages contain work
products and feasibility evidence, such as prototypes, studies, estimates, bases of estimates.
After reviewing the package and any other evidence, risks are determined. If the risks are
acceptable or negligible, commit to the next phase. If the risks are high, but probably
addressable, do more work within the current phase. If the risks are too high or unaddressable,
then efforts should be discontinued altogether.
2.13
Lean Risk Management Plan
The word “Plan” often conjures up a heavyweight document full of boilerplate and hard-toremember sections. Risk management plans should be particularly lean and risk-driven as
illustrated in the outline presented in Figure 2-5.
Risk Management Plan Outline
1. Objectives (the “Why”)
2. Deliverables and Milestones (the “What” and “When”)
3. Responsibilities (the “Who” and “Where”)
4. Approach (the “How”)
5. Resources (the “How Much”)
Figure 2-5. Lean Risk Management Plan Outline
The following is an example of how the outline might be used to develop a risk management
plan to conduct fault tolerance prototyping to mitigate identified risks that the fault tolerance
features might seriously compromise performance aspects such as throughput, real-time deadline
satisfaction, and power consumption. It also shows the examples of the types of information to
be provided for responsibilities, the risk management approach, and the needed funding
resources.
Objectives: Determine, reduce level of risk of the fault tolerance features causing unacceptable
performance (e.g., throughput, response time, power consumption). Create a description of and a
development plan for a set of low-risk fault tolerance features
Deliverables and Milestones:

By week 3: Evaluation of fault tolerance option, assessment of reusable components, draft
workload characterization, evaluation plan for prototype exercise, and description of
prototype.
106753919
- 36 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5

By week 7: Operational prototype with key fault tolerance features; workload simulation;
instrumentation and data reduction capabilities; and draft description, plan for fault tolerance
features.

By week 10: Evaluation and iteration of prototype; revised description, plan for fault
tolerance features.
Responsibilities:

System Engineer: G. Smith—Tasks 1, 3, 4, 9, 11, support of tasks 5, 10

Lead Programmer: C. Lee—Tasks 5, 6, 7, 10 support of tasks 1, 3

Programmer: J. Wilson—Tasks 2, 8, support of tasks 5, 6, 7, 10
Approach: Design-to-Schedule prototyping effort; driven by hypotheses about fault toleranceperformance effects; using multicore processor, real-time OS, add prototype fault tolerance
features. Evaluate performance with respect to representative workload. Refine Prototype based
on results observed.
Resources:
$60K
Full-time system engineer, lead programmer, programmer (10 weeks)*(3
staff)*($2K/staff-week)
$0K
3 Dedicated workstations (from project pool)
$0K
2 Target processors (from project pool)
$0K
1 Test co-processor (from project pool)
$10K
Contingencies
$70K
Total
Note that much of this risk plan information can be provided in a resource-loaded network tool.
2.14
FED Development Process Framework
The appropriate level of detail for the contents of the FED are based on the perceived risks and
criticality of the system to be developed. It is NOT a “one size fits all” process, but rather a
framework to help developers and stakeholders determine the appropriate level of analysis and
evaluation. It is also not limited to new system developments, but can be effective in major
upgrades to existing systems or systems of systems.
As with other ICM artifacts, FED process and content are risk-driven. Generic sets of steps are
provided, but they need to be tailored to each situation. Tailoring occurs at increasing levels of
detail in Exploration, Validation, and Foundations phases. Tailoring can be satisfied by pointers
to existing evidence. Tailoring also applies to the Stage II Foundations rebaselining process
Within the FED, representative scenarios, including critical off-noinal scenarios, should be
provided for exercising large simulations, and testbed evaluation facilities, processes and
evaluation criteria should also be prepared.
106753919
- 37 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
2.15
Version 0.5
Steps for Developing Feasibility Evidence
Table 2-1 outlines a process that can be used for developing feasibility evidence. The process
clearly depends on having the appropriate work products for the phase (operational concept,
requirements specification, architecture, etc.). As part of the engineering work, critical
feasibility assurance issues are identified that are critical to the success of the system
development program. These are the issues for which options are explored, as well as potentially
viable options, further investigated with respect to cost, schedule, Return on Investment (ROI),
etc.
106753919
- 38 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
Table 2-1. Steps for Developing FED
Step
Description
Examples/Detail
A.
Develop phase work-products/artifacts
See ICM Anchor Point Milestone Content
charts provided in the chapter for each
phase/anchor point milestone.
B.
Determine most critical feasibility
assurance issues
Issues for which lack of feasibility evidence is
program-critical
C.
Evaluate feasibility assessment
options
Cost-effectiveness, risk reduction
leverage/ROI, rework avoidance
Tool, data, scenario availability
D.
Select options, develop feasibility
assessment plans
E.
Prepare FED assessment plans and
earned value milestones
Try to relate earned value to risk-exposure
avoided rather than budgeted cost
F.
Begin monitoring progress with
respect to plans
Also monitor project/technology/objectives changes and
adapt plans
G.
Prepare evidence-generation enablers
Assessment criteria
Parametric models, parameter values, bases of
estimate
COTS assessment criteria and plans
Benchmarking candidates, test cases
Prototypes/simulations, evaluation plans,
subjects, and scenarios
Instrumentation, data analysis capabilities
H.
Perform pilot assessments; evaluate
and iterate plans and enablers
I.
Assess readiness for Commitment
Review
J.
Hold Commitment Review when
ready; adjust plans based on review
outcomes
Shortfalls identified as risks and covered by
risk mitigation plans
Proceed to Commitment Review if ready
NOTE: “Steps” denoted by letters rather than numbers to indicate that many are done
concurrently.
Also note that the FED activities need to be planned and assigned an appropriate earned value.
The earned value should be based on the potential risk exposure costs, not the perceived
106753919
- 39 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
available budget. See Competitive Prototyping (Chapter 5) for additional guidance on how much
“information” to buy via the feasibility assessment process.
2.16
Large-Scale Simulation and Testbed FED Preparation Example
As seen in Figure 2-6, for large, complex systems, it is often very cost effective to begin
development of a modeling and simulation testbed to support both the system infrastructure and
application developers as well as those trying to evaluate the feasibility of the various
alternatives.
Figure 2-6. Large-Scale Simulation and Testbed FED Preparation Example
106753919
- 40 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
2.17
Version 0.5
Example of FED Risk Evaluation Criteria
In order to evaluate the findings of a feasibility assessment and determine the appropriate next
steps, criteria are needed to help determine the associated risk for each finding. The following
information can be used to assign a risk level to the various findings and guide the outcome of
the associated Commitment Review:
Negligible: Anticipated 0-5% budget and/or schedule overrun. Identified only minor shortfalls
and imperfections expected to affect the delivered system.
Low: Anticipated 5-10% budget and/or schedule overrun. Identified 1-3 moderate shortfalls and
imperfections expected to affect the delivered system.
Moderate: Anticipated 10-25% budget and/or schedule overrun. Identified >3 moderate
shortfalls and imperfections expected to affect the delivered system.
Major: Anticipated 25-50% budget and/or schedule overrun. Identified 1-3 mission-critical
shortfalls and imperfections expected to affect the delivered system.
Severe: Anticipated >50% budget and/or schedule overrun. Identified >3 mission-critical
shortfalls and imperfections expected to affect the delivered system.
2.18
Conclusions
Anchor Point milestones enable synchronization and stabilization of concurrent engineering.
They have been successfully applied on small to large projects. APs also provide incremental
stakeholder resource commitment points. The FED enables evidence of program feasibility to be
evaluated at each of the APs and are produced by developer and then evaluated by the
stakeholders and independent experts. To successfully develop a FED for each AP, the FED
needs to be defined as a “first class” deliverable, it needs to be planned (with resources), and part
of the program earned value management system. Any shortfalls in evidence are sources of
uncertainty and risk, and should be covered by risk management plans. And finally, the FED
and associated risk management plan(s) should be part of the traditional milestone content and
reviews.
2.19
References
B. Boehm and W. Hansen, “The Spiral Model as a Tool for Evolutionary Acquisition,”
CrossTalk, May 2001.
B. Boehm, A.W. Brown, V. Basili, and R. Turner, “Spiral Acquisition of Software-Intensive
Systems of Systems,” CrossTalk, May 2004, pp. 4-9.
B. Boehm and J. Lane, “Using the ICM to Integrate System Acquisition, Systems Engineering,
and Software Engineering,” CrossTalk, October 2007, pp. 4-9.
J. Maranzano et. al., “Architecture Reviews: Practice and Experience,” IEEE Software,
March/April 2005.
R. Pew and A. Mavor, Human-System Integration in the System Development Process: A New
Look, National Academy Press, 2007.
106753919
- 41 -
Version Date: 12/31/08
Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs
Version 0.5
W. Royce, Software Project Management, Addison Wesley, 1998.
RQ-4A/B Global Hawk High Altitude, Long Endurance, Unmanned Reconnaissance Aircraft,
USA, http://www.airforce-technology.com/projects/global/, accessed on 8 July 2008.
R. Valerdi, “The Constructive Systems Engineering Cost Model,” Ph.D. dissertation, USC,
August 2005.
CrossTalk articles: http://www.stsc.hill.af.mil/crosstalk
106753919
- 42 -
Version Date: 12/31/08
Chapter 3: ICM Fundamentals – Incremental Adoptability
Version 0.5
3Chapter 3: ICM Fundamentals – Incremental
Adoptability
3.1
Introduction
Most system development programs are not new developments, but are currently focused on
maintaining existing capabilities, enhancing existing capabilities, or incorporating new
capabilities. For these existing programs, it is often not possible to make major process changes
all at once. Rather, it is better to incrementally change processes over time.
Existing programs may benefit from some ICM principles and practices, but not others. Problem
programs may find some ICM practices helpful in recovering viability. There are primary
opportunities for incremental adoption of ICM principles and practices, this chapter deals with
many of them.
3.2
Supplementing Traditional Requirements and Design Reviews with
Development and Review of Feasibility Evidence
[Further elaboration TBD]
3.3
Stabilized Incremental Development and Concurrent Architecture
Rebaselining
[Further elaboration TBD]
3.4
Using Schedule as Independent Variable and Prioritizing Features to
be Delivered
[Further elaboration TBD]
3.5
Continuous Verification and Validation
[Further elaboration TBD]
3.6
Using the Process Decision Table
[Further elaboration TBD]
106753919
- 43 -
Version Date: 12/31/08
Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase
Version 0.5
4Chapter 4: Stage I: Incremental Definition –
The Needs and Opportunities/Exploration
Phase
4.1
Introduction
The ICM is the only current life cycle process model that follows DoDI 5000.02 in beginning
with User Needs and Technology Opportunities. Programs using process models that start with
pinning down Requirements often run into trouble by overfocusing on requirements for materiel
users and overlooking other success-critical stakeholders (such as maintainers and interoperators)
and their needs; by prematurely imposing requirements constraints that disqualify superior
solution opportunities (such as commercial-off-the-shelf (COTS) or contractor capabilities, as in
the Section 2-3 example); or by quickly converging on requirements that lead to obsolete
solutions, by neglecting growth trends in user needs or by basing requirements on technologies
with limited growth potential (such as inability to accommodate workload growth or system-ofsystems interoperability).
Figure 4-1 shows the nature and location of the Needs and Opportunities phase in the DoD life
cycle. The main objective of this phase is to develop evidence that enables the Milestone
Decision Authority to correctly decide whether a materiel solution is appropriate to pursue. This
is done in the ICM based on the level of risk involved in making the decision to proceed, as
shown in the “Risk?” circle at the bottom of the Needs and Opportunities phase column in Figure
4-1. If the evidence indicates an acceptable level of risk in going forward toward a materiel
solution, the upward-right arrow passes the project into the next Materiel Solution Analysis and
Analysis of Alternatives phase.
If the evidence points to a clear decision that a non-materiel solution is best, the downward arrow
is followed and the materiel solution process is discontinued, or perhaps rescoped to explore an
alternative mission scope for which a materiel solution may be useful and appropriate. If the
evidence to support a clear decision is insufficient, the upward-left arrow is taken and further
exploration of needs and opportunities is pursued. For very small systems for which an alreadyowned COTS product can be tailored to produce the solution, the risk of going directly into
Development is negligible, and the right-pointing arrows can be taken to skip phases, but this is
not the case for major defense acquisition programs (MDAPs).
For MDAPs, a non-trivial level of investment by several involved stakeholders is needed to
produce the required evidence to support a Materiel Development Decision. These stakeholders
generally want some level of evidence that this investment will be worthwhile for them. Thus,
the Needs and Opportunities phase begins with a similar evidence/risk-based decision to proceed
into the Needs and Opportunities phase, based on evidence provided by the Service or Agency
desiring to pursue a materiel-development acquisition process.
106753919
- 44 -
Version Date: 12/31/08
Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase
Version 0.5
Figure 4-1. Overview of the ICM - Exploration Highlighted
The primary Activities to be concurrently performed in a risk- and opportunity-driven process in
the Needs and Opportunities phase (and later phases) are shown in Figure 4-2. This figure shows
that Understanding Needs and Envisioning Opportunities are key activities to be performed at
the beginning of the DoD life cycle process. Other key activities shown in Figure 4-2 are
exploring the level of need and scoping of a materiel solution; the identification and
prioritization of its goals and objectives; the characterization of its legacy system or
interoperability constraints; the architectural compatibility of its candidate hardware, software,
and human factors elements; and its affordability based on early life-cycle planning and
estimating efforts. Evidence of the feasibility of the candidate materiel solution is provided by
some combination of prototyping, modeling, simulation, benchmarking, and cost-schedule
analysis, and used to enable stakeholders to assess their risks of proceeding and to negotiate
mutual commitments to invest further resources in proceeding or continuing to develop adequate
evidence of feasibility.
106753919
- 45 -
Version Date: 12/31/08
Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase
Version 0.5
Figure 4-2. ICM Activity Categories and Level of Effort
4.2
Phase Key Elements
Table 4-1 identifies the key phase elements in the Needs and Opportunities phase. Each element
will be elaborated in a section below.
106753919
- 46 -
Version Date: 12/31/08
Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase
Version 0.5
Table 4-1. Needs and Opportunities (Exploration)
Goals & Objectives
 Prepare for
MDD
 Develop ICD:
OpCon, Needed
Capabilities,
MSR
 Identify SCSHs
 Identify
Common-Case
or alternative
life cycle
process
 Evidence of
Feasibility, Risk
Addressal
 Next-phase plans
 SCSH
commitment to
way forward
Entry
Conditions
 Committed
protagonist
 Phase plan
 Adequate
resources to
achieve
goals and
objectives:
funds, time,
staff, and
info sources
 Fit to
enterprise
strategy
Inputs
Steps (concurrent)
Exit Conditions
Work Products
Pitfalls
 Top-level
 Understand needs:
 Validated shared
 MSR: Evidence
 Inadequate phase
phase plan
and
rationale
 Funds,
time, staff,
info
sources
 SCSH
support
for phase
current and future gaps,
DOTMLPF aspects
 Explore opportunities
 Identify enabling
initiatives and SCSHs
 Determine SCSH
objectives and priorities
 Identify feasible
vision package:
Benefits chain &
SCSHs; System
boundary, OpCon;
Key solution
elements; and FED
(including business
case)
 Validated MSR,
approved ICD
solution region
 Evaluate MDSs, nonMDSs
 Identify Common-Case
or alternative life cycle
process
 Develop and validate
shared vision elements
 Prepare MSR, ICD
 Obtain SCSH decision
commitments for next
phase
 Common-Case or
alternative life cycle
process
 Feasibility Evidence or
risk-management
plans
 CP strategy or
candidate alternatives
to analyze
 Detailed next-phase
plans
 Committed to by
that needs are
critical; non-MDS
solutions are
inadequate; at
least one MDS is
feasible
 Top-level shared
vision package
 Associated risk
mitigation plans
 Common-Case or
alternative life
cycle process
 SCSH
Commitment
MOAs
 Identification of
next-level
SCSHs, IPT
structure
 Detailed nextphase plans, ICD
plans, budget and
schedule
 Neglecting SCSHs
 Shortfalls in FEDs
or SCSH MOAs
without riskmanagement plans
 Poor project fit to
enterprise strategy
 Overconstrained
and incompatible
quality factors
 Shortfalls in CP
strategy
 Cursory tech
readiness analysis,
analysis of nonMDS inadequacy
 Inadequate nextphase plans,
budget, schedule
SCSHs
CP = Competitive Prototyping
DOTMLPF = Doctrine, Organization Training, Materiel, Leadership and education, Personnel, Facilities
FED = Feasibility Evidence Description
ICD = Initial Capabilities Document
MDD = Materiel Development Decision
MDS = Materiel Development Solution
MOA = Memorandum of Agreement
MSR = Materiel Solution Rationale
OpCon = Operational Concept
SCSH = Success-Critical Stakeholder
106753919
- 47 -
Version Date: 12/31/08
Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase
4.3
Version 0.5
Goals and Objectives
4.3.1
Prepare for Material Development Decision
4.3.2
Develop Initial Capabilities Document
4.3.3
Identify the Success-Critical Stakeholders
4.3.4
Identify Common-Case or Alternative Life Cycle Process
4.3.5
Evidence of Feasibility, Risk Addressal
4.3.6
Next-Phase Plans
4.3.7
SCSH Commitment to Way Forward
4.4
Entry Conditions
4.4.1
Committed Protagonist
4.4.2
Phase Plan
4.4.3
Adequate Resources to Achieve Goals and Objectives
4.4.4
Fit to Enterprise Strategy
4.5
Inputs
4.5.1
Top-Level Phase Plan and Rationale
4.5.2
Funds, Time, Staff, Info Sources
4.5.3
Success-Critical Stakeholder Support for Phase
4.6
Steps (Concurrent)
4.6.1
Understand Needs
4.6.2
Explore Opportunities
4.6.3
Identify Enabling Initiatives and Success-Critical Stakeholders
106753919
- 48 -
Version Date: 12/31/08
Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase
Version 0.5
4.6.4
Determine Success-Critical Stakeholder Objectives and Priorities
4.6.5
Identify Feasible Solution Region
4.6.6
Evaluate Materiel Development Solutions, Non-Materiel
Development Solutions
4.6.7
Identify Common-Case or Alternative Life Cycle Process
4.6.8
Develop and Validate Shared Vision Elements
4.6.9
Prepare Materiel Solution Rationale, Initial Capabilities
Document
4.6.10
Obtain Success-Critical Stakeholder Decision Commitments for
Next Phase
4.7
Exit Conditions
4.7.1
Validated Shared Vision Package
4.7.2
Validated Materiel Solution Rationale, Approved Initial
Capabilities Document
4.7.3
Common-Case or Alternative Life Cycle Process
4.7.4
Feasibility Evidence or Risk-Management Plans
4.7.5
CP Strategy or Candidate Alternatives to Analyze
4.7.6
Detailed Next-Phase Plans
4.7.7
Committed to by Success-Critical Stakeholders
4.8
Work Products
4.8.1
Materiel Solution Rationale
4.8.2
Top-Level Shared Vision Package
4.8.3
Associated Risk Mitigation Plans
4.8.4
Common-Case or Alternative Life Cycle Process
106753919
- 49 -
Version Date: 12/31/08
Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase
Version 0.5
4.8.5
Success-Critical Stakeholder Commitment Memoranda of
Agreement
4.8.6
Identification of Next-Level Success-Critical Stakeholders,
Integrated Product Team Structure
4.8.7
Detailed Next-Phase Plans, Initial Capabilities Document
4.9
Pitfalls
4.9.1
Inadequate Phase Plans, Budget and Schedule
4.9.2
Neglecting Success-Critical Stakeholders
4.9.3
Shortfalls in Feasibility Evidence Descriptions or Success-Critical
Stakeholder Memoranda of Agreements Without RiskManagement Plans
4.9.4
Poor Project Fit to Enterprise Strategy
4.9.5
Overconstrained and Incompatible Quality Factors
4.9.6
Shortfalls in Competitive Prototyping Strategy
4.9.7
Cursory Technical Readiness Analysis, Analysis of Non-Materiel
Development Solution Inadequacy
4.9.8
Inadequate Next-Phase Plans, Budget, Schedules
106753919
- 50 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
Version 0.5
5Chapter 5: Stage I: Incremental Definition –
The ICM Process Decision Table
5.1
Introduction
As with other models trying to address a wide variety of situations, the general form of the
Incremental Commitment Model is rather complex. However, its risk-driven nature enables
projects to determine a set of twelve common risk patterns and organize them into a decision
table that can help new projects converge on a process that fits well with their particular process
drivers. For each of these twelve special cases, the ICM Process Decision Table provides toplevel guidelines for tailoring the key activities of the ICM, along with suggested lengths between
each internal system build and each external system increment delivery.
The process drivers used as inputs to the Decision Table include: the system’s size and
complexity, its rate of change, its mission-criticality, the extent of non-developmental item
(NDI) support for its desired capabilities, and the capability of available organizational and
personnel for developing the system.
The Decision Table includes twelve common risk-driven special cases of the ICM. With
representative examples in parentheses, these are (1) Use NDI (Small accounting system), (2)
Agile (Small e-services), (3) Scrum of Scrums (Business data processing), (4) Formal methods
(LSI chip, security kernel), (5) Software-embedded hardware component (Multi-sensor control
device), (6) Indivisible initial operational capability (Complete vehicle platform), (7) NDIintensive system (Supply chain management), (8) Hybrid agile/plan-driven system (C4ISR
system), (9) Multi-owner system of systems (SoS) (Net-centric emergency services), (10) Family
of systems (Medical device product line), (11) Brownfield (obsolete C4ISR system replacement),
and (12) Net-Centric Services (a: community support; b: quick response decision support).
In addition, some of the larger special cases can be architected to enable some of their
components to be treated more simply as one of the smaller special cases. Often, the top-level
choice of special case can be made by the end of the ICM Needs and Opportunities/Exploration
Phase, with the lower-level special case decisions made in subsequent phases.
5.2
The ICM Process Decision Table
In many situations, the explorations in the Needs and Opportunities phase discussed in Chapter 4
will show that a special-case process opportunity exists that will satisfy the users’ needs without
the full set of phases and activities in ICM. These cases cover the very small to the very large as
well as the use of commercial off-the-shelf (COTS) software products to the development of a
large, complex custom software application or integrated sets of software applications. Each of
these situations presents risks at the various stages of development, some risks more critical than
others. The goal of the ICM is to identify these risks and then tailor the process to include rigor
where necessary to investigate and manage the risks and to streamline the process when risks are
negligible, allowing the development team to be more agile when possible. Table 5-1 contains a
list of the special cases of the ICM and an example of each case. These cases cover a broad
spectrum of systems. In addition, Table 1 describes the characteristics of the case (or application
106753919
- 51 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
Version 0.5
category) with respect to size and complexity; expected change rate during the development
process; the overall criticality of the application, typically as it applies to human life,
security/protection of information, or financial liability; and the type of NDI support one might
expect for an application in this category. It also indicates the organizational and personally
capabilities that are required to be successful at building this type of application and the
typical/suggested lengths for each internal system build and each external system increment
delivery. Table 5-2 provides top-level guidelines for tailoring the key activities in Stage I
(incremental definition) and Stage II (incremental development and operations) of the ICM.
106753919
- 52 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
Version 0.5
2. Agile
1-30
Low-Med
Business data
processing
Security
kernel;
Safety-critical
LSI chip
Multi-sensor
control device
Med
1-10
Med-High
Low
0.3
Extra
High
Low
0.3-1
Complete
vehicle
platform
Supply chain
management
MedHigh
8. Hybrid agile/
plan-driven
system
9. Multi-owner
system of systems
5. HW with
embedded SW
component
6. Indivisible
IOC
7. NDI- intensive
10. Family of
systems
11. Brownfield
12a. Net- Centric
Services—
Community
Support
Time/Build;
Time/Increment
Complete
Low
3. Architected
Agile
4. Formal
Methods
Organizational
and Personnel
Capability
Small
accounting
E-services
NDI Support
1. Use NDI
Criticality
Example
Change Rate
(%/Month)
Special Case
Size,
Complexity
Table 5-1. Characteristics of the Risk-Driven Special Cases of the ICM
Good; in
place
Good; most
in place
None
Agile-ready
Med-high
Agile-ready
Med-high
Strong formal
methods
experience
<= 1 day;
2-6 weeks
2-4 weeks;
2-6 months
1-5 days;
1-4 weeks
Med-Very
High
Good; in
place
Experienced;
med-high
SW: 1-5 days;
Market-driven
0.3-1
HighVery High
Some in
place
Experienced;
med-high
MedHigh
0.3-3
Med-Very
High
NDI-driven
architecture
C4ISR system
MedVery
High
Mixed
parts;
1-10
Mixed parts
Net-centric
military
operations
Medical
device
product line
Incremental
legacy
phaseout
Community
Services or
Special
Interest
Group
Response to
competitor
initiative
Very
High
Mixed
parts;
1-10
1-3
Mixed
parts;
Med-Very
High
Very High
NDIexperienced;
med-high
Mixed parts
SW: 2-6 weeks;
Platform: 6-18
months
SW: 1-4 weeks;
Systems: 6-18
months
1-2 months;
9-18 months
MedVery
High
HighVery
High
LowMed
Med-Very
High
0.3-3
Med-High
0.3-3
Low-Med
Many NDIs;
some in
place
Some in
place
NDI as
legacy
replacement
Tailorable
service
elements
Related
experience,
med-high
Related
experience,
med-high
Legacy reengineering
NDIexperienced
2-4 months;
18-24 months
1-2 months;
9-18 months
2-6 weeks/
refactor ;
2-6 months
<= 1 day;
6-12 months
12b. Net-Centric
Med3-30
Med-High Tailorable
NDI<= 1 day ;
Services—Quick
High
service
experienced
QR-driven
Response
elements
Decision
Suppport
C4ISR: Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance; HW: Hardware;
IOC: Initial Operational Capability; NDI: Non-Development Item; QR : Quick Response ; SW: Software.
106753919
- 53 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
Version 0.5
Table 5-2. Key Activities for Each Special Case of the ICM
Special Case
Example
1. Use NDI
Small
accounting
E-services
2. Agile
3. Architected Agile
Business data
processing
4. Formal Methods
Security kernel;
Safety-critical
LSI chip
Multi-sensor
control device
5. HW with
embedded SW
component
6. Indivisible IOC
Complete
vehicle
platform
7. NDI- intensive
Supply chain
management
8. Hybrid agile/
plan-driven system
C4ISR system
9. Multi-owner
system of systems
Net-centric
military
operations
Medical device
product line
10. Family of
systems
11. Brownfield
12a. Net-Centric
Services—
Community Support
Incremental
legacy phaseout
Community
Services or
Special Interest
Group
Response to
competitor
initiative
Key Stage I Activities:
Incremental Definition
Acquire NDI
Key Stage II Activities:
Incremental Development,
Operations
Use NDI
Skip Valuation, Architecting
phases
Combination Valuation,
Architecting phases. Complete
NDI preparation
Precise formal specification
Scrum plus agile methods of
choice
Architecture-based Scrum of
Scrums
Concurrent HW/SW
engineering. CDR-level ICM
DCR
Determine minimum-IOC
likely, conservative cost. Add
deferrable SW features as risk
reserve
Thorough NDI-suite life cycle
cost-benefit analysis, selection,
concurrent requirements/
architecture definition
Full ICM; encapsulated agile in
high change, low-medium
criticality parts (Often HMI,
external interfaces)
Full ICM; extensive multiowner team building,
negotiation
Full ICM; Full stakeholder
participation in product line
scoping. Strong business case
Re-engineer/refactor legacy into
services
Filter, select, compose, tailor
NDI
IOC Development, LRIP, FRP.
Concurrent version N+1
engineering
Drop deferrable features to meet
conservative cost. Strong award
fee for features not dropped
Formally-based programming
language; formal verification
Pro-active NDI evolution
influencing, NDI upgrade
synchronization
Full ICM, three-team
incremental development,
concurrent V&V, nextincrement rebaselining
Full ICM; large ongoing
system/software engineering
effort
Full ICM. Extra resources for
first system, version control,
multi-stakeholder support
Incremental legacy phaseout
Evolve tailoring to meet
community needs
12b. Net-Centric
Filer, select, compose, tailor
Satisfy quick response; evolve or
Services—Quick
NDI
phase out
Response Decision
Suppport
C4ISR: Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance;
CDR: Critical Design Review DCR: Development Commitment Review FRP: Full-Rate Production
HMI: Human-Machine Interface HW: Hardware IOC: Initial Operational Capability
LRIP: Low-Rate Initial Production NDI: Non-Development Item SW: Software
V&V: Verification and Validation.
106753919
- 54 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
5.3
Version 0.5
Using the Process Decision Table
Selecting and Tailoring a Decision Table Case. The general ICM chart has many decision
options available, but its risk-driven approach results in most projects having simpler special
cases that can usually be determined during the project’s Exploration phase. The decision table
indicates the most common special cases. But it is not exhaustive. If your risk pattern is not
included, look at the closest approximation to it, and think through an appropriate risk-driven
variant of it.
Even if your pattern is included in the table, there may be special aspects of your project that
require additional tailoring (e.g., multiple cultures, distant time zones, legacy constraints). The
decision table should be a stimulus to thinking, not a substitute for it. In addition, many complex
projects have simpler pieces in them. Rather than apply a one-size-fits-all process to all of them,
often you can architect the system to use simpler processes on the simpler parts.
The rest of this section describes each of the ICM special cases and typical risks associated with
each case.
5.3.1
Case 1: Use NDI (Non-Development Items)
Suppose you have an application for which an appropriate NDI (COTS, open source, reuse
library, customer-furnished package) solution is available, and other options of developing
perhaps a better version yourself or outsourcing such a development. Even if you produce a
better solution (frequently not the case), you will generally incur more expense and take longer
to begin to capitalize on its benefits. And you will often find that the NDI package has features
that you hadn’t realized you would need, but are there when you need them.
On the other hand, there are risks that may disqualify some NDIs. They may be overly complex
for your needs, incompatible with your other applications, or highly volatile. See the discussion
in Section 33.1 of Software Engineering Economics (Boehm, 1981) for more about the pros and
cons of NDI solutions.
5.3.2
Case 2: Pure Agile Methods
If your project is small (less than 10 people) and its criticality involves the loss of discretionary
vs. essential funds, a pure agile method such as Extreme Programming (Beck, 1999), Scrum
(Schwaber, 2002), or Crystal (Cockburn, 2002) is generally best if you have relatively highcapability, agile-ready personnel. The risks of using a more formal, plan-driven approach are
less adaptability to change and belated feedback on the product’s capabilities. The biggest risk is
to try to develop the application all at once for several months, and then find that it is a mismatch
to the users’ needs. Agile developers have found that the best way to address this risk is to
organize the project into short (2-6 week) delivery increments that may be incomplete, but
provide early useful capabilities. Any flaws can then be detected early and fixed in the next
increment (for very high criticality applications, this would not be acceptable). On a small
project it is also easy to set up a daily build and regression test structure that identifies
integration problems early when they are easier to fix.
Some risks of using the agile approach is that it is harder to write a contract for what is being
developed; that it may sub optimize for early success by using unscalable capabilities (fourth106753919
- 55 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
Version 0.5
generation languages, running all in main memory); or high personnel turnover. See (Boehm
and Turner, 2004), pp. 121-128 for an example risk analysis of an agent-based event planning
system, ending with a decision to go agile.
5.3.3
Case 3: Architected Agile
For medium-size (20-80 people), medium complexity (reasonably mature and scalable
technology; largely compatible shareholders), agile methods can be scaled using an Architected
Agile approach with early investment in a largely change-prescient architecture and
user/developer/customer team building. For relatively stable projects (0.3-1% change/month),
plan-driven methods can be used with low risk. But for higher rates of changes (1-10%/month),
a more agile approach is less risky. A risk analysis of a 50-person, medium sized architecturebased agile supply chain management project is provided on pages 106-121 of (Boehm and
Turner, 2004). A number of organizations in such areas as corporate infrastructure, medical,
aerospace, and ERP applications have reported significant gains in adaptability and quality of the
Architected Agile approach over plan-driven methods for such projects. However, others that
had less capable and agile-ready people, less management and customer commitment, and less
up-front architecture investment have not. (Boehm, 2007)
5.3.4
Case 4: Formal Methods
Formal methods involve the development and verification of a precise mathematical
specification of the behavior of a hardware and/of software systems; an implementation of the
system in formal-semantics-based languages; and a mathematical proof that the implementation
is exactly equivalent to the specification (no more; no less). Such methods are expensive relative
to standard commercial practice and require scarce high-capability personnel, but are
inexpensive relative to a massive product recall, expensive lawsuits, or a massive loss of
valuable and confidential information.
Current formal methods generally require repeating the expensive mathematical proof process
whenever the specification or implementation is changed, making the approach less viable for
systems with highly volatile requirements. In general, non-developmental items are not precisely
specified and verified enough to be safely used in such systems. Also, formal methods have
limited scalability; almost all fully-verified software systems have less than 10,000 lines of code.
However, some progress is being made toward modularizing such systems so that
implementations and proofs can be built up incrementally via lower-level lemmas and theorems.
5.3.5
Case 5: Software Embedded Hardware Component
The application classes above have been mostly software-intensive. The differences in economic
and risk patterns for hardware-intensive projects (Boehm and Lane, 2007) will create different
risk-based special cases of the ICM. Once a project commits to a particular manufacturing
approach and infrastructure, the hardware cost of change will be much higher. And the primary
costs at risk are in development and manufacturing, particularly if the component is cheaper to
replace than to fix or if fixes can be accomplished by software workarounds. Thus, the ICM
Stage I activities of producing and validating detailed specifications and plans are much more
cost-effective than for the agile cases above. They will often go beyond the usual level of detail
(Critical Design Review versus evidence-based Preliminary Design Review) for an ICM
Development Commitment Review (DCR), since so many of the details are likely to be major
106753919
- 56 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
Version 0.5
sources of rework expenditures and delay if wrong. And after Initial Operational Capability
(IOC) development, the rework risks generally dictate an incremental low rate initial production
phase before a full-rate production phase in Stage II. In case the component is planned to evolve
to subsequent hardware-software reconfigurations, the ICM approach of having a concurrent
systems engineering team developing specifications and plans for the “N+1st “ increment could
be adopted. The length of these later increments will be driven by the product’s marketplace or
competitive situation.
5.3.6
Case 6: Indivisible IOC
More complex hardware intensive systems, such as aircraft, may have a great deal of software
that can be incrementally developed and tested, but may have an indivisible hardware IOC that
must be developed as a unit before it can be safely tested (e.g., an automobile or aircraft’s
braking system or full set of safety-critical vehicle controls). Relative to the cost and duration of
the software increments, the indivisible hardware IOC has two significant sources of risk:

It cannot fit into smaller software increments

It cannot drop required hardware features to meet IOC schedule/cost/quality as independent
variable.
The first can be addressed by synchronizing the hardware IOC with the Nth software increment
(e.g., (Rechtin and Maier, 1997) osculating orbits for hardware and software).
If some combination of schedule/cost/quality is truly the project IOC’s independent variable,
then one does not want to commit to a most-likely combination of schedule/cost/quality, as there
will be a roughly 50% chance of an overrun or quality shortfall in completing the indivisible
IOC. Rather, it is better to determine a conservative IOC cost and schedule for meeting the
quality objective, and use the difference between the conservative cost and schedule and the
most-likely cost and schedule as a risk reserve. The best way to do this is to use the conservative
cost and schedule as the IOC target, and to determine a set of desired but non-essential, easy to
drop software features that could be developed within the risk reserve. Then, if the most-likely
indivisible IOC capability begins to overrun, some desired software features can be dropped
without missing the scheduled delivery time and cost. There should also be a strong award-fee
incentive for the developers to minimize the number of features that need to be dropped.
5.3.7
Case 7: NDI-Intensive
Our experiences in developing USC web-service applications between 1996 and 2004 was that
they went from 28% of the application’s functionality being delivered by NDI components to
80% (Yang et al, 2005). A similar trend was identified by the 2001 Standish Report, which
reported that 53% of the functionality of commercial software applications was being delivered
by NDI components in 2000 (Standish, 2001). The economics of NDI-intensive systems dictates
a bottom-up versus a top-down approach to system development, in which the capability
envelope of the NDI determines the affordable requirements, rather than a top-down
requirements-to-capability approach. A large supply-chain management system may need to
choose among several NDI candidates each for such functions as inventory control, trend
analysis, supplier/customer relations management, transportation planning, manufacturing
control, and financial transactions; and evaluate not only the candidates’ cost/performance
aspects, but also their interoperability with each other and with the corporation’s legacy
106753919
- 57 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
Version 0.5
infrastructure. Besides NDI assessment, other significant sources of effort can be NDI tailoring,
NDI integration, and effect of NDI version volatility and obsolescence; see (Yang et al, 2005).
A particular challenge in Stage II is the effect of NDI volatility and obsolescence. Surveys have
indicated that commercial NDI products have a new release about every 10 months, and that old
releases are supported by the vendor for about 3 releases. Some large systems have had about
120 NDI components, indicating that about 12 components will have new releases each month,
and that not upgrading will leave each component unsupported in about 30 months. In such
cases, a great deal of attention needs to be paid to upgrade synchronization, and to pro-active
NDI evolution influencing. Some large organizations synchronize their NDI upgrades to their
major re-training cycles of about 12-18 months. For additional NDI best practices, see (Wallnau
et al, 2002).
5.3.8
Case 8: Hybrid Agile/Plan-Driven System
Some large, user-intensive systems such as C4ISR systems, air traffic control systems, and
network control systems have a mix of relatively stable, high-criticality elements (sensors and
communications; key business logic) and more volatile, more moderately critical elements such
as GUI displays, electronic warfare countermeasures, and interfaces with externally evolving
systems. As many acquisitions of this nature have shown, it is highly risky to apply one-sizefits-all processes and incentive structures to these differing classes of elements. Instead, in Stage
I, it is important to determine which system elements belong in which class along with the other
functions of understanding needs, envisioning opportunities, NDI assessment, scoping the
system, and determining feasible requirements and increments performed for complex systems in
Stage I; to architect the system to encapsulate the volatile parts for development by agile teams;
and to organize to use plan-driven development for the other elements of the overall system. In
Stage II, the cycles for the agile teams are likely to be significantly shorter than those for the
plan-driven teams.
5.3.9
Case 9: Multi-Owner System of System
In this situation, your goal is to integrate a set of existing systems (or guide and evolve the
integration of a set of existing systems). These systems are primarily developed, owned, and
maintained by an organization other than the one that is attempting to manage and guide the set
of systems as a system of systems. Because of the independence of these constituent systems,
the SoS organization has little or no formal control over the processes used to maintain and
evolve the constituent systems. The SoS may be an enterprise wide business SoS, with the
constituents being primarily COTS products along with some legacy applications. Or it may be
Department of Defense (DoD) warfighting SoS, where the constituent legacy systems are
integrated to increase capabilities on the battle field.
Traditional systems engineering (SE) activities are typically tailored for the SoS case to define an
SoS architecture, better coordinate the activities of multiple systems in migrating to the SoS
architecture, and provide synchronization points for the SoS. In (OUSD AT&L, 2008), pilot
studies have shown that many DoD SoS have re-organized the traditional SE activities into a set
of seven core elements: 1) translating capability objectives, 2) understanding systems and
relationships, 3) assessing performance to capability objectives, 4) developing, evolving, and
maintaining SoS design, 5) monitoring and assessing changes, 6) addressing new requirements
106753919
- 58 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
Version 0.5
and options, and 7) orchestrating updates to SoS. Further analysis shows, that these elements
map fairly well to the hybrid agile/plan-driven case (Case 8) at the SoS level.
What makes this case different from Case 8, is that each of the constituent systems is using their
own processes, which could be any of the above cases, depending on the scope, complexity, and
characteristics of the constituent system. What is key is that the Case 9 extends Case 8 to
include information or participation of the constituent systems in the agile planning activities and
lets the “battle rhythm” of the constituent system increments guide the SoS plan-driven and
V&V activities.
5.3.10
Case 10: Family of Systems
Families of systems are typically a set of systems that belong to a product line and can be easily
used to customize a solution for a given need. This might be a suite of medical devices or a suite
of applications to customer support. This is often the set of systems developed by a vendor that
become the NDI components for CASE 7 above. Again, the rigor required for the SoS case is
present here. However, in this situation, the family of systems is typically owned and evolved by
a single organization/vendor and presents a case where the owning organization has much more
control over the evolution of the components of the family of systems, thus possibly reducing
some risks and allowing the ICM process to be a little more streamlined.
5.3.11
Case 11: Brownfield
A Brownfield counterexample involved a major US corporation that used a Greenfield systems
engineering and development approach to develop a new Central Corporate Financial System to
replace a patched-together collection of strongly-coupled and poorly documented COBOL
business data processing programs. The system included an early Enterprise Resource Planning
(ERP) system that was well matched to the corporation's overall financial needs, but not to its
detailed business processes, which included various workarounds to compensate for difficulties
with the legacy software. The new system was well organized to support incremental
implementation, but was dropped at a cost of $40 million after two failed tries to provide
continuity of service, due primarily to the infeasibility of incrementally phasing out the legacy
software and business processes compatibly with the new system's incremental capabilities.
The ICM can be used to organize systems engineering and acquisition processes in ways that
better support Brownfield legacy software re-engineering so that organizations can better provide
continuity of services. In particular, its concurrent activities of Understanding Needs,
Envisioning Opportunities, System Scoping and Architecting, Feasibility Evidence
Development, and Risk/Opportunity Assessment enable projects to focus specifically on their
legacy system constraints and on opportunities to deal with them.
The application of the ICM to the Brownfield corporate counterexample situation would have
avoided the failures of the Greenfield development. Its Understanding Needs activity would
have determined the ways in which the legacy system had intertwined financial and other
business services. For examples, Project Services included budgeting and scheduling, work
breakdown system accounting, earned value management intertwined with requirements, version
and configuration management; Contract Services included expenditure category management,
billing, and receivables management intertwined with deliverables management and engineering
106753919
- 59 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
Version 0.5
change proposal tracking; with similar intertwining in Personnel Services and Marketing
Services.
The ICM's Envisioning Opportunities activity would have identified opportunities to use largescale refactoring methods to decouple the financial and non-financial elements of these services
in ways that would make it feasible for them to interoperate with a Financial Services element.
Its System Scoping and Architecting activity would have repackaged the legacy software and
developed the new architecture around financial services that would support incremental
phaseout, and its Feasibility Evidence Development activity would have assessed any
outstanding risks and covered them with risk management plans that would be tracked to ensure
project success.
A good example of a Brownfield methodology with several case study examples is provided in
(Hopkins and Jenkins, 2008).
5.3.12
Case 12a: Net-Centric Services—Community Support
Net-Centric Services for community support tend to fall into two categories. One involves
community service organizations providing needed services for child care, elder care,
handicapped, homeless, or jobless people. Their information processing service needs include
tracking of clients, volunteers, donors, donations, events, and provision of news and
communication services. These used to require considerable programming to realize, but now
can be realized by combining and tailoring NDI packages offering such services. The other
category of net-centric services for community support involves special interest groups
(professionals, hobbyists, committees, ethnic groups) with similar needs for tracking members,
interests, subgroups, events, and provision of news and communication services. Development
of such capabilities involves much more prototyping and much less documentation than is
needed for more programming-oriented applications; for example, the internals of the NDI
packages are generally unavailable for architectural documentation, and the resulting system
capabilities are more driven by NDI capabilities than by prespecified requirements documents.
A good example of how the ICM provides support for such net-centric services is provided in
(Boehm and Bhuta, 2008) and the entire journal provides additional approaches and case studies
of net-centric services development.
5.3.13
Case 12b: Net-Centric Services: Quick Response Decision
Support
Another form of net-centric services involves rapidly configuring a capability to analyze
alternative decision options in response to a competitor's initiative. These might involve a
competitor beginning to penetrate a business's home territory, and might involve the need to
evaluate the relative costs and benefits of alternative strategies of expanding services, expanding
service locations, or repricing products or services. NDI packages to help analyze geographical,
logistical, human resources, and financial implications of alternative competitive response
decisions can be rapidly configured to provide a quick-response capability for converging on a
decision. Once the decision is made, the resulting capability may be phased out, or evolved into
a more general and maintainable capability for similar future situations. Again, the
November/December 2008 special issue of IEEE Software on "Opportunistic Software Systems
106753919
- 60 -
Version Date: 12/31/08
Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table
Version 0.5
Development" (Boehm and Bhuta, 2008) provides additional perspectives on this area of netcentric services development.
5.4
References
Beck, K. 1999. Extreme programming explained, Addison Wesley.
Boehm, B. 1981. Software engineering economics, Prentice Hall.
Boehm, B. 2007. Agility and quality”, Keynote Presentation, ICSE 2007 Workshop on Software
Quality.
Boehm, B. and Bhuta, J. 2008. Balancing Risks and Opportunities in Component-Based
Software Development. IEEE Software, November/December 2008, pp. 56-63.
Boehm, B. and Lane, J. 2007. Using the incremental commitment model to integrate system
acquisition, systems engineering, and software engineering, CrossTalk, Vol. 20, No. 10.
Boehm, B. and Turner, R. 2004. Balancing agility and discipline: a guide for the perplexed.
Addison-Wesley, Boston.
Cockburn, A. 2002. Agile software development, Addison Wesley.
Hopkins, R. and Jenkins, K. 2008. Eating the IT Elephant: Moving from Greenfield
Development to Brownfield, IBM Press, 2008
Marenzano, J. et al. 2005. Architecture reviews: practice and experience. IEEE Software,
March/April 2005, pp. 34-43.
Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (OUSD
AT&L), 2008. Systems engineering for systems of systems, Version 1.0. Washington, DC:
Pentagon.
Pew, R. W., and Mavor, A. S. 2007. Human-System Integration in the System Development
Process: A New Look. National Academy Press.
Schwaber, K. and Beedle, M. 2002. Agile Software Development with Scrum. Prentice Hall.
Standish Group 2001. Extreme Chaos.
Wallnau, K., Hissam, S., and Seacord, R. 2002. Building Systems from Commercial Components.
Addison Wesley.
106753919
- 61 -
Version Date: 12/31/08
Chapter 6: Stage I: Incremental Definition – Materiel Solution Analysis & AoA/Valuation Phase
Version 0.5
6Chapter 6: Stage I: Incremental Definition –
The Materiel Solution Analysis &
AoA/Valuation Phase
6.1
Introduction
If the Needs and Opportunities phase leads to a positive Materiel Development Decision, the
project enters into the DoD Materiel Solution Analysis & Analysis of Alternatives phase.
Section 4 of DoDI 5000.02 Enclosure 2 summarizes the system acquisition guidelines for this
phase. Its primary objective is to satisfy the criteria for passing Milestone A. These criteria
include performing analysis of at least two alternative Materiel Solution approaches, and
providing evidence that at least one alternative is feasible to pursue, at least through the next
phase of Technology Development and documentation of the systems Capability Development
approach.
Figure 6-1. Overview of the ICM – MSA and AoA Highlighted
6.2
Phase Key Elements
Table 6-1 identifies the key phase elements in the Materiel Solution Analysis; Analysis of
Alternatives phase. Each element will be elaborated in a section below.
106753919
- 62 -
Version Date: 12/31/08
Chapter 6: Stage I: Incremental Definition – The Materiel Solution Analysis & AoA/Valuation Phase
Version 0.5
Table 6-1 Materiel Solution Analysis; Analysis of Alternatives (Valuation)
Goals &
Objectives
Entry
Conditions
Inputs
Steps (concurrent)
Exit Conditions
Work
Products
Pitfalls







Analysis of
Alternatives
with FED,
detailed
business case
Key risks
resolved or
covered by
risk
mitigation
plans
Key
infrastructure
selected or
underway
Technology
Development
Strategy
(TDS)
Next-phase
plans
SCSH
commitment
to way
forward
Inadequate phase
budget and schedule

 Neglecting SCSHs,
particularly external


interoperators


 Neglecting
alternatives

 Easiest-first

approaches

 Prototyping


shortfalls,


especially in
scalability and off
nominal cases

 Delays in FED,

business-case

preparation

 Shortfalls in FEDs,
SCSH MOAs



without risk

mitigation plans


 Shortfalls in CP
strategy
 Inadequate nextphase plans, budget,
schedule
CP = Competitive Prototyping
FED = Feasibility Evidence Description
IPT = Integrated Product Team
MOE = Measures of Effectiveness
MOA = Memorandum of Agreement
OpCon = Operational Concept
RAA = Responsibility, Authority, Accountability
SCSH = Success-Critical Stakeholder
TDS = Technology Development Strategy
106753919
Needs and
Opportunities
phase exit
conditions
Well-staffed
and resourced
IPTs
Key risk and
infrastructure
plans
Alternatives
analysis or CP
preparation
plans: criteria,
scenarios,
testbeds for
needs
satisfaction,
critical
technology
maturity, life
cycle costs,
manufacturing
feasibility
Needs and
Opportunities
phase work
products
Alternatives
analysis/CP &
infrastructure
determination
plans, staff,
resources
SCSH support
for phase
Elaborate IPT
structure, RAAs,
plans, and
staffing and/or
CP strategy/plans
Establish crossIPT work rhythm
and coordination
Refine and
execute phase
and CP plans;
monitor progress
versus plans
Monitor changes
in needs,
opportunities,
and risks
Adapt plans to
needed change
Select CPs;
evaluate results
Develop and
validate TDS
- 63 -
Validated TDS:
OpCon,
prototypes,
requirements,
architecture,
system and
increment plans,
feasibility/business
-case MOE
evidence or risk
plans
Key infrastructure
selected or
underway
Detailed nextphase plans,
including CP
strategy, staffing,
Cost and Software
Data Reporting
Committed to by
SCSHs
Validated
TDS
Prototyping
of high-risk
elements,
system
structure,
and
behavior
Key
support and
infrastructu
re
selections
for next
phase
Detailed
next-phase
plans
SCSH
Commitme
nt MOAs
Version Date: 12/31/08
Chapter 6: Stage I: Incremental Definition – Materiel Solution Analysis & AoA/Valuation Phase
6.3
Version 0.5
Goals and Objectives
6.3.1
Analysis of Alternatives with Feasibility Evidence Description,
Detailed Business Case
6.3.2
Key Risks Resolved or Covered by Risk Mitigation Plans
6.3.3
Key Infrastructure Selected or Underway
6.3.4
Technology Development Strategy
6.3.5
Next-Phase Plans
6.3.6
Success-Critical Stakeholder Commitment to Way Forward
6.4
Entry Conditions
6.4.1
Needs and Opportunities Phase Exit Conditions
6.4.2
Well-Staffed and Resourced Integrated Product Teams
6.4.3
Key Risk and Infrastructure Plans
6.4.4
Alternatives Analysis or Competitive Prototyping Preparation
Plans
6.5
Inputs
6.5.1
Needs and Opportunities Phase Work Products
6.5.2
Alternatives Analysis / Competitive Prototyping and
Infrastructure Determination Plans, Staff, Resources
6.5.3
Success-Critical Stakeholder Support for Phase
6.6
Steps (Concurrent)
6.6.1
106753919
Elaborate Integrated Product Team Structure; Responsibility,
Authority, Accountabilities; Plans, and Staffing and/or
Competitive Prototyping Strategy/Plans
- 64 -
Version Date: 12/31/08
Chapter 6: Stage I: Incremental Definition – Materiel Solution Analysis & AoA/Valuation Phase
Version 0.5
6.6.2
Establish Cross-Integrated Product Team Work Rhythm and
Coordination
6.6.3
Refine and Execute Phase and Competitive Prototyping Plans;
Monitor Progress Versus Plans
6.6.4
Monitor Changes in Needs, Opportunities, and Risks
6.6.5
Adapt Plans to Needed Change
6.6.6
Select Competitive Prototypes; Evaluate Results
6.6.7
Develop and Validate Technology Development Strategy
6.7
Exit Conditions
6.7.1
Validated Technology Development Strategy
6.7.2
Key Infrastructure Selected or Underway
6.7.3
Detailed Next-Phase Plans, Including Competitive Prototyping
Strategy, Staffing, Cost and Software Data Reporting
6.7.4
Committed to by Success-Critical Stakeholders
6.8
Work Products
6.8.1
Validated Technology Development Strategy
6.8.2
Prototyping of High-Risk Elements, System Structure, and
Behavior
6.8.3
Key Support and Infrastructure Selections for Next Phase
6.8.4
Detailed Next-Phase Plans
6.8.5
Success-Critical Stakeholder Commitment Memoranda of
Agreement
6.9
Pitfalls
6.9.1
106753919
Inadequate Phase Budget and Schedule
- 65 -
Version Date: 12/31/08
Chapter 6: Stage I: Incremental Definition – Materiel Solution Analysis & AoA/Valuation Phase
Version 0.5
6.9.2
Neglecting Success-Critical Stakeholders, Particularly External
Interoperators
6.9.3
Neglecting Alternatives
6.9.4
Easiest-First Approaches
6.9.5
Prototyping Shortfalls, Especially in Scalability and Off-Nominal
Cases
6.9.6
Delays in Feasibility Evidence Description, Business-Case
Preparation
6.9.7
Shortfalls in Feasibility Evidence Descriptions, Success-Critical
Stakeholder Memoranda of Agreement Without Risk Mitigation
Plans
6.9.8
Shortfalls in Competitive Prototyping strategy
6.9.9
Inadequate Next-Phase Plans, Budget, Schedule
106753919
- 66 -
Version Date: 12/31/08
Chapter 7: Stage I: Incremental Definition –The Technology Development & CDD/Foundations Phase
Version 0.5
7Chapter 7: Stage I: Incremental Definition –
The Technology Development &
CDD/Foundations Phase
7.1
Introduction
The objective of the Technology Development and Capability Development Document phase is
to ensure that the proposed acquisition program has fully prepared its technical and management
foundations for passing Milestone B, initiating an acquisition program, and entering the
Engineering and Manufacturing Development phase. These foundations include not only
prototypes and other evidence of technology maturity, but also a fully developed system
architecture and associated feasibility evidence sufficient to pass FED-based Preliminary Design
Review (now scheduled before Milestone B in DoDI 5000.02), along with a fully elaborated
Capability Development Document, with validated plans and feasibility evidence for cost and
schedule estimates.
This phase begins at the Anchor Point (AP) Foundations Commitment Review (FCR), and the
passage of DoD Milestone A, as seen in Figure 7-1.
106753919
- 67 -
Version Date: 12/31/08
Chapter 7: Stage I: Incremental Definition –The Technology Development & CDD/Foundations Phase
Version 0.5
Figure 7-1. Overview of the ICM – Technology Development and CDD Highlighted
Although the title of Section 5 of DoDI 5000.02’s Enclosure 2 is “Technology Development
Phase,” its content includes the full set of technical and management foundations necessary for a
successful Engineering and Manufacturing Development phase. Figure 7-2 shows the results of
a root cause analysis of negative findings on over 50 DoD program assessments, which identified
a number of risk sources that are not generally covered by the development and evaluation of
technology. Overfocus on technology development without having good management processes,
acquisition practices, requirements processes, and shortfalls in staffing, organizing, and
contracting will lead to similar negative program findings.
106753919
- 68 -
Version Date: 12/31/08
Chapter 7: Stage I: Incremental Definition –The Technology Development & CDD/Foundations Phase
Version 0.5
Figure 7-2. Categorization of DoD Program Assessment Negative Findings
Some of the root causes, such as verification and validation (V&V) shortfalls, competing
priorities, lack of appropriate staff, and program realism, have been root causes of prototyping
and technology maturity shortfalls, but causes of other shortfalls as well. There are good ways to
address these additional risk sources within the scope of the new DoDI 5000.02 by placing
stronger emphasis on such documents as the Technology Development Strategy and Capability
Development Document, and on providing evidence of program achievability as well as
technology maturity at Milestone B. As discussed below, the ICM provides process support for
the concurrent development and validation of the risk-driven set of technology and management
foundations.
7.2
Phase Key Elements
Table 7-1 identifies the key phase elements in the Technology Development and Capabilities
Development Document phase. Each element will be elaborated in a section below.
106753919
- 69 -
Version Date: 12/31/08
Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase
Version 0.5
Table 7-1. Technology Development and Capability Development Document: TD-CDD (Foundations)
Goals &
Objectives
Entry
Conditions
Inputs
Steps (concurrent)




Exit Conditions
Work
Products
Pitfalls
Ensure
 Validated CDD
 Complete,
 Inadequate phase
technology
complete for
validated
budgets,
readiness for
increment 1
CDD for
schedules
mission needs
increment 1  Neglecting


 Later features
 Establish
prioritized and
 Validated
SCSHs
baseline EMD
initially allocated
acquisition
 Prototyping
increment
to later increments,
strategy
shortfalls
content, budgets,
including
and
 Overemphasis on
and schedules
production,
baseline

technology: also

sustainment plans,
content for
 Prepare for,
need refined
ready technology
later

execute CP
requirements,
and infrastructure,
increments,
strategy

validated designs,
FED
production

 Develop and
cost estimates,
 Realistic risk
 Feasibility

validate CDD:
manufacturing

mitigation plans
evidence,
OpCon,
processes, infor remaining risks
realistic
acquisition
place
risk
strategy, feature
 EMD and
infrastructure,
mitigation
prioritization and
production
acquisition
plans for
allocation to
acquisition
management,
remaining

increments,
strategy
staff capabilities
risks
preliminary and
 Committed to by
 SCSH
detailed design,
 SCSH
SCSHs
commitments
development,
Commitme  Inadequate teamintegration and
nt MOAs
building
test, feasibility
 Unprioritized
evidence
features
 Obtain decision
 Inadequate nextcommitments for
phase budgets,
EMD increment
schedules
1
AoA = Analysis of Alternatives
CDD = Capability Development Document
CP = Competitive Prototyping
EMD = Engineering and Manufacturing Development
FED = Feasibility Evidence Description
KPP = Key Performance Parameter
MOA = Memorandum of Agreement
MSA = Materiel Solution Analysis
OpCon = Operational Concept
SCSH = Success-Critical Stakeholder
TD = Technology Development
TDS = Technology Development Strategy
Full TD-CDD
with missionready
technology,
infrastructure,
FED, life
cycle and
increment
plans
Risks resolved
or covered by
risk plans
Worked-out
supplier and
strategicpartner
participation
plans and
MOAs
SCSH
commitment
to life-cycle
plans and
approach
106753919
MSA/AoA
phase exit
conditions
Clear scoping
of
recommended
-alternative
objectives,
constraints,
and priorities
Desired and
acceptable
KPP levels
Approved
TDS:
Technology
development,
CP, risk
resolution and
infrastructure
plans, staff,
resources
MSA/AoA
phase work
products
Recommende
d-alternative
objectives,
constraints,
and priorities
Desired and
acceptable
KPP levels
Approved
TDS
Necessary
staffing,
budgets,
schedules,
info sources
- 70 -
Version Date: 12/31/08
Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase Version 0.5
7.3
Goals and Objectives
7.3.1
Full Technology Development and Capabilities Development
Document with Mission-Ready Technology, Infrastructure,
Feasibility Evidence Description, Life Cycle and Increment Plans
7.3.2
Risks Resolved or Covered by Risk Plans
7.3.3
Worked-Out Supplier and Strategic-Partner Participation Plans
and Memoranda of Agreement
7.3.4
Success-Critical Stakeholder Commitment to Life-Cycle Plans
and Approach
7.4
Entry Conditions
7.4.1
Materiel Solution Analysis / Analysis of Alternatives Phase Exit
Conditions
7.4.2
Clear Scoping of Recommended-Alternative Objectives,
Constraints, and Priorities
7.4.3
Desired and Acceptable Key Performance Parameter Levels
7.4.4
Approved Technology Development Strategy
7.5
Inputs
7.5.1
Materiel Solution Analysis / Analysis of Alternatives Phase Work
Products
7.5.2
Recommended-Alternative Objectives, Constraints, and Priorities
7.5.3
Desired and Acceptable Key Performance Parameter Levels
7.5.4
Approved Technology Development Strategy
7.5.5
Necessary Staffing, Budgets, Schedules, Info Sources
7.6
106753919
Steps (Concurrent)
- 71 -
Version Date: 12/31/08
Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase Version 0.5
7.6.1
Ensure Technology Readiness for Mission Needs
7.6.2
Establish Baseline Engineering and Manufacturing Development
Increment Content, Budgets, and Schedules
7.6.3
Prepare For and Execute Competitive Prototyping Strategy
7.6.4
Develop and Validate Capability Development Development
7.6.5
Obtain Decision Commitments for Engineering and
Manufacturing Development Increment 1
7.7
Exit Conditions
7.7.1
Validated Capability Development Document Complete for
Increment 1
7.7.2
Later Features Prioritized and Initially Allocated to Later
Increments, Including Production, Sustainment Plans, Ready
Technology and Infrastructure, Feasibility Evidence Document
7.7.3
Realistic Risk Mitigation Plans for Remaining Risks
7.7.4
Engineering and Manufacturing Development and Production
Acquisition Strategy
7.7.5
Committed to by Success-Critical Stakeholders
7.8
Work Products
7.8.1
Complete, Validated Capability Development Document for
Increment 1
7.8.2
Validated Acquisition Strategy and Baseline Content for Later
Increments, Production
7.8.3
Feasibility Evidence, Realistic Risk Mitigation Plans for
Remaining Risks
7.8.4
Success-Critical Stakeholder Commitment Memoranda of
Agreement
106753919
- 72 -
Version Date: 12/31/08
Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase Version 0.5
7.9
Pitfalls
7.9.1
Inadequate Phase Budgets and Schedules
7.9.2
Neglecting Success-Critical Stakeholders
7.9.3
Prototyping Shortfalls
7.9.4
Overemphasis on Technology
7.9.5
Success-Critical Stakeholders Commitments
7.9.6
Inadequate Team-Building
7.9.7
Unprioritized Features
7.9.8
Inadequate Next-Phase Plans, Budget, Schedule
106753919
- 73 -
Version Date: 12/31/08
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V
Version 0.5
8Chapter 8: Stage II: Incremental
Developments – Engineering and
Manufacturing Development and Beyond
8.1
Introduction
As shown in the overall DoD-ICM life cycle diagram in Figure 8-1, Stage II covers not only
Engineering and Manufacturing Development (EMD), but also how EMD increments are
synchronized with activities beyond such as Production & Deployment and Operations &
Maintenance. As discussed in the Executive Summary and Chapter 1 on ICM Principles, current
and future trends toward emergent requirements and rapid change make incremental,
evolutionary development the most cost-effective approach for most current and future DoD
systems.
Figure 8-1. Overview of the Generic Incremental Commitment Life Cycle Process – Stage
II Highlighted
106753919
- 74 -
Version Date: 12/31/08
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V
Version 0.5
DoD mission-critical systems must also have high levels of security, safety, reliability,
availability, and maintainability, making agile development methods less viable. However, they
may have less-critical components for which agile methods may be preferable. This Chapter will
focus on Major Defense Acquisition Programs (MDAPs), but will include a section on special
cases such as agile methods for less-critical, software-intensive systems of subsystems. Other
special cases will include ICM variants for systems of systems, and for systems in which several
software increments may fit within a longer hardware increment.
Next, this chapter will continue with a more detailed discussion than in the Executive Summary
of the overall structure of and motivation for the ICM Stage II process. It will then elaborate a
representative Major Defense Acquisition Program (MDAP) increment in the previous format of
Goals & Objectives, Preconditions, Inputs, Steps, Postconditions, Work Products, and Pitfalls.
Finally, this chapter will address the special cases discussed above.
8.1.1
ICM Stage II Structure and Motivation
Figure 8-2 shows a single increment of the ICM Stage II process. It assumes that the project has
developed in Stage I and continues to evolve:

A best-effort definition of the system’s long-range capability;

An incremental sequence of prioritized capabilities culminating in the long-range capability;

A Feasibility Rationale providing sufficient evidence that the system architecture will
support the incremental capabilities, that each increment can be developed within its
available budget and schedule, and that the series of increments create a satisfactory return
on investment for the organization and mutually satisfactory outcomes for the successcritical stakeholders.
106753919
- 75 -
Version Date: 12/31/08
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V
Version 0.5
Figure 8-2. ICM Stage II: Increment Activities
As seen in Figure 8-2, the model is organized to simultaneously address the conflicting 21st
century DoD challenges of rapid change and high assurance of dependability. It also addresses
the need for rapid fielding of incremental capabilities with a minimum of rework, and the other
major 21st DoD century trends involving integration of systems and software engineering, COTS
components, legacy systems, net-centric services, and systems of systems..
Rapid change in DoD systems comes in two primary forms. One is relatively predictable change
that can be handled by the plan-driven Parnas strategy of encapsulating sources of change within
modules, so that the effects of changes are largely confined to individual modules. The other is
relatively unpredictable change that may appear simple (such as adding a “cancel” or “undo”
capability to the software or adding system payload components to the hardware), but often
requires a great deal of agile adaptability to rebaseline the architecture and incremental
capabilities into a feasible solution set.
The need to deliver high-assurance incremental capabilities on short fixed schedules means that
each increment needs to be kept as stable as possible. This is particularly the case for very large
systems of systems with deep supplier hierarchies (often 6 to 10 levels), in which a high level of
rebaselining traffic can easily lead to chaos. In keeping with the use of the ICM as a risk-driven
process model generator, the risks of destabilizing the development process make this portion of
the project into a build-to-specification approach in which unforeseeable changes are deferred.
The need for high assurance of each increment also makes it cost-effective to invest in a team of
appropriately skilled personnel to continuously verify and validate the increment as it is being
developed.
106753919
- 76 -
Version Date: 12/31/08
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V
Version 0.5
However, “deferring the change traffic” does not imply deferring its change impact analysis,
change negotiation, and rebaselining until the beginning of the next increment. With a single
development team and rapid rates of change, this would require a team optimized to develop to
stable plans and specifications to spend much of the next increment’s scarce calendar time
performing tasks much better suited to agile teams. Thus, it is important not to disband the
systems engineering team at Milestone B, but to keep a critical mass of agile systems
engineering talent on the project to handle the change traffic and rebaseline the plan,
specifications, and feasibility rationales for the remaining increments.
8.1.2
Agile Rebaselining and the C2ISR Metaphor
The appropriate metaphor for these tasks is not a build-to-specification metaphor or a
purchasing-agent metaphor but an adaptive “command-control-intelligence-surveillancereconnaissance” (C2ISR) metaphor. It involves an agile team performing the first three activities
of the C2ISR “Observe, Orient, Decide, Act” (OODA) loop for the next increments, while the
plan-driven development team is performing the “Act” activity for the current increment.
“Observing” involves monitoring changes in relevant technology and COTS products, in the
competitive marketplace, in external interoperating systems, and in the environment; and
monitoring progress on the current increment to identify slowdowns and likely scope deferrals.
“Orienting” involves performing change impact analyses, risk analyses, and tradeoff analyses to
assess candidate rebaselining options for the upcoming increments. “Deciding” involves
stakeholder renegotiation of the content of upcoming increments, architecture rebaselining, and
the degree of COTS upgrading to be done to prepare for the next increment. It also involves
updating the future increments’ Feasibility Rationales to ensure that their renegotiated scopes
and solutions can be achieved within their budgets and schedules and without compromising
performance.
A successful rebaseline means that the plan-driven development team can hit the ground running
at the beginning of the “Act” phase of developing the next increment, and the agile team can hit
the ground running on rebaselining definitions of the increments beyond. Figure 8-3 shows how
this three-team cycle (lean, plan-driven, stabilized developers; thorough V&Vers; agile, proactive rebaseliners) plays out from one increment to the next.
As much as possible, usage feedback from the previous increment (e.g., Operations and
Support1) is not allowed to destabilize the current increment (e.g., EMD, PDR2), but is fed into
the definition of the upcoming development increment (e.g., TD, CDD, LCSP3). Of course,
some level of mission-critical updates will need to be fed into the current increment, but only
when the risk of not doing so is greater that the risk of destabilizing the current increment.
Stage I: Incremental Definition | Stage II: Incremental Development, Production & Ops.
106753919
- 77 -
Version Date: 12/31/08
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V
Version 0.5
Figure 8-3. DoD Version of ICM for Evolutionary Development
Again, in keeping with the use of the spiral model as a risk-driven process model generator, the
risks of not adequately and rapidly adapting the next increments to address the sources of needed
change mean that this portion of the project is not a build-to-specification waterfall process.
Instead, it is a risk-driven set of concurrent prototyping, analysis, and stakeholder renegotiation
activities leading to a best-possible redefinition of the plans and specifications to be used by the
stabilized development team for the next increment.
8.2
Phase Key Elements
Table 8-1 identifies the key phase elements in the Engineering and Manufacturing Developmentn
phase. Each element is elaborated in a section below.
106753919
- 78 -
Version Date: 12/31/08
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V
Version 0.5
Table 8-1. Engineering and Manufacturing Developmentn (Developmentn)
Goals &
Objectives
Entry
Conditions
Inputs
Steps (concurrent)
Exit Conditions
Work
Products
 Develop, V&V,
and transition
the nth
increment of
capability
 Deliver on
schedule, defer
low-priority
features if
necessary
 Prepare
rebaselined
specifications,
plans, FED for
incrementn+1
 Execute next
phase of
manufacturing
plans
 Acceptable
responses to
change requests
 SCSH
commitment to
life-cycle plans
and approach
 TD-CDDn exit
conditions
 Stabilized and
prioritized
requirements,
specifications,
and plans
 Adequate
staffing;
funding of
development,
rebaselining,
and V&V
teams; and
manufacturing
capabilities
 TD-CDDn work
products
 Incrementn
development
and V&V plans
 IPT, risk
resolution,
infrastructure,
plans, staff,
resources
 Change traffic
from previousincrement users,
management,
technology, and
competition
 Stabilized build-tospecifications
development of
incrementn
 Continuous V&V
of build-tospecifications
artifacts
 Next-increments
rebaselining, FEDs
based on change
traffic inputs
 Development
progress
monitoring and
scope adjustment
 Incrementn
installation,
operational test,
training, and
acceptance
 Incrementn
manufacturing and
test preparations
 Accepted incrementn
operational
capability
 Associated
DOTMLPF
capabilities
 Satisfactory
disposition of
change traffic
 Validated,
rebaselined nextincrements CDDs
 Incrementn
manufacturing and
test plans and
preparations
 Committed to by
SCSHs
 Accepted
incrementn
operational
capability
 Associated
DOTMLPF
capabilities
 Validated,
rebaselined
nextincrements
CDDs
 Incrementn
manufacturin
g and test
plans and
preparations
 SCSH
Commitment
MOAs
Pitfalls
 Inadequate phase
budgets, schedules
 Neglecting SCSHs
 Destabilizing
incrementn
development
 Inadequate
development
monitoring and
rescoping
 Inadequate test and
transition plans and
preparations
 Inadequate changesource monitoring
and response
 Unvalidated and
unprioritized nextincrement
capabilities
 Weak
manufacturing
process and quality
controls
 Inadequate nextphase budgets,
schedules
CDD = Capability Development Document
DOTMLPF = Doctrine, Organization Training, Materiel, Leadership and education, Personnel, Facilities
FED = Feasibility Evidence Description
IPT = Integrated Product Team
MDD = Materiel Development Decision
MOA = Memorandum of Agreement
OpCon = Operational Concept
SCSH = Success-Critical Stakeholder
TD = Technology Development
V&V = Verification and Validation
106753919
- 79 -
Version Date: 12/31/08
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V
8.3
Version 0.5
Goals and Objectives
8.3.1
Develop, Verification and Validation, and Transition the nth
Increment of Capability
8.3.2
Deliver on Schedule, Defer Low-Priority Features if Necessary
8.3.3
Prepare Rebaselined Specifications, Plans, Feasibility Evidence
Document for Incrementn+1
8.3.4
Acceptable Responses to Change Requests
8.3.5
Success-Critical Stakeholder Commitment to Life-Cycle Plans
and Approach
8.4
Entry Conditions
8.4.1
Technology Development-Capability Development Documentn
Exit Conditions
8.4.2
Stabilized and Prioritized Requirements, Specifications, and Plans
8.4.3
Adequate Staffing; Funding of Development, Rebaselining, and
Verification and Validation Teams; and Manufacturing
Capabilities
8.5
Inputs
8.5.1
Technology Development-Capability Development Documentn
Work Products
8.5.2
Incrementn Development and Verification and Validation Plans
8.5.3
Integrated Product Team, Risk Resolution, Infrastructure, Plans,
Staff, Resources
8.5.4
Change Traffic from Previous-Increment Users, Management,
Technology, and Competition
8.6
106753919
Steps (Concurrent)
- 80 -
Version Date: 12/31/08
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V
Version 0.5
8.6.1
Stabilized Build-to-Specifications Development of Incrementn
8.6.2
Continuous Verification and Validation of Build-to-Specifications
Artifacts
8.6.3
Next-Increments Rebaselining, Feasibility Evidence Descriptions
Based on Change Traffic Inputs
8.6.4
Development Progress Monitoring and Scope Adjustment
8.6.5
Incrementn Installation, Operational Test, Training, and
Acceptance
8.6.6
Incrementn Manufacturing and Test Preparations
8.7
Exit Conditions
8.7.1
Accepted Incrementn Operational Capability
8.7.2
Associated Doctrine, Organization Training, Materiel, Leadership
and Education, Personnel, Facilities Capabilities
8.7.3
Satisfactory Disposition of Change Traffic
8.7.4
Validated, Rebaselined Next-Increments Capability Development
Documents
8.7.5
Incrementn Manufacturing and Test Plans and Preparations
8.8
Work Products
8.8.1
Accepted Incrementn Operational Capability
8.8.2
Associated Doctrine, Organization Training, Materiel, Leadership
and Education, Personnel, Facilities Capabilities
8.8.3
Validated, Rebaselined Next-Increments Capability Development
Documents
8.8.4
Incrementn Manufacturing and Test Plans and Preparations
8.8.5
Success-Critical Stakeholder Commitment Memoranda of
Agreement
106753919
- 81 -
Version Date: 12/31/08
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V
8.9
Version 0.5
Pitfalls
8.9.1
Inadequate Phase Budgets and Schedules
8.9.2
Neglecting Success-Critical Stakeholders
8.9.3
Destabilizing Incrementn Development
8.9.4
Inadequate Development Monitoring and Rescoping
8.9.5
Inadequate Test and Transition Plans and Preparations
8.9.6
Inadequate Change-Source Monitoring and Response
8.9.7
Unvalidated and Unprioritized Next-Increment Capabilities
8.9.8
Weak Manufacturing Process and Quality Controls
8.9.9
Inadequate Next-Phase Plans, Budget, Schedule
8.10
Special Cases
The agile, systems of systems, and hardware-software increment synchronization special case
discussions will go here.
Figure 8-4. Example of SoSE Coordination Challenge
Note: Chapter 9 may be merged into Chapter 8, or it may continue to be a separate chapter.
106753919
- 82 -
Version Date: 12/31/08
Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V
Version 0.5
Acknowledgements
We would like to thank our sponsors in the OUSD(AT&L) Systems and Software Engineering
Directorate, Kristen Baldwin and Bruce Amato, for their strategic guidance of this effort. We
have also benefited greatly from the technical discussions on the ICM and its application to DoD
projects with Elliot Axelband (Rand/USC), Winsor Brown (USC), Judith Dahmann (Mitre),
Edgar Dalrymple (US Army), Chuck Driessnack (SAIC/MDA), Marilynn Goo (Boeing), Gary
Hafen (LMCO), Peter Hantos (Aerospace), Dan Ingold (USC), Sue Koolmanojwong (USC), Ray
Madachy (NPS), Art Pyster (Stevens), Rick Selby (NGC), Bob Skalamera (OSD/Stevens), Rich
Turner (Stevens), Gan Wang (BAE Systems), and the members of the NRC study on HumanSystem Integration in the System Development Process. We owe a special degree of thanks to
Alfred Brown for an outstanding job of technical editing.
106753919
- 83 -
Version Date: 12/31/08