Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects Version 0.5 Barry Boehm Jo Ann Lane December 31, 2008 106753919 i Version Date: 12/31/08 Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects Version 0.5 Version History Date Author Version Changes made Rationale 12/31/08 BB, JAL 0.5 First draft created. Create first draft. 106753919 ii Version Date: 12/31/08 Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects Version 0.5 Table of Contents GUIDE FOR USING THE INCREMENTAL COMMITMENT MODEL (ICM) FOR SYSTEMS ENGINEERING OF DOD PROJECTS............................................................ I VERSION 0.5 ................................................................................................................... I VERSION HISTORY ...................................................................................................... II TABLE OF CONTENTS ................................................................................................. III TABLE OF TABLES ..................................................................................................... VII TABLE OF FIGURES .................................................................................................. VIII 0 Executive Summary....................................................................................................... 1 0.1 Abstract ............................................................................................................ 1 0.2 Purpose and Scope ............................................................................................ 1 0.3 Motivation and Context: Current DoD Needs and Future Challenges ................. 2 0.4 The 6 Key Systems Engineering Process Principles ............................................ 4 0.5 Overview of the Generic ICM and DoD Life Cycle Processes ............................. 5 0.6 ICM Metaphor and Competitive Prototyping Example ...................................... 9 0.7 The DoD Version of the ICM ........................................................................... 11 0.8 What is Being Concurrently Engineered in the ICM? ...................................... 13 0.9 How is All This Concurrent Engineering Synchronized and Stabilized? ........... 15 0.10 Project Experience with ICM Principles .......................................................... 17 0.11 Structure and Content of the Guide ................................................................. 18 0.12 Conclusions ..................................................................................................... 21 0.13 References....................................................................................................... 21 1 Chapter 1: ICM Fundamentals – Fundamental Principles ............................................ 24 1.1 Introduction .................................................................................................... 24 1.2 Key Principle #1: Commitment and Accountability of System Engineers, Developers, and Managers ........................................................................................ 24 1.3 Key Principle #2: Stakeholder Satisficing......................................................... 24 1.4 Key Principle #3: Incremental and Evolutionary Growth of System Definition and Stakeholder Commitment .................................................................................. 24 106753919 iii Version Date: 12/31/08 Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects Version 0.5 1.5 Key Principle #4: Iterative System Development and Definition ....................... 25 1.6 Key Principle #5: Concurrent System Definition and Development .................. 25 1.7 Key Principle #6: Evidence-Based, Risk Driven Decision Milestones ................ 25 1.8 References....................................................................................................... 25 2 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs ..................................... 26 2.1 Developing and Using Feasibility Evidence for Life Cycle Commitment Milestones ................................................................................................................ 26 2.2 Nature of FEDs and AP Milestones .................................................................. 26 2.3 Problems Encountered without FED................................................................ 27 2.4 Problems Avoidable with FED ......................................................................... 28 2.5 AT&T Experience with AP Reviews ................................................................ 29 2.6 Overview of ICM Phases, Anchor Points, and Decision Options ....................... 30 2.7 Key Point: Need to Show Evidence ................................................................. 31 2.8 Off-Nominal Architecture-Breakers ................................................................ 31 2.9 Common Examples of Inadequate Evidence..................................................... 32 2.10 Examples of Making the Evidence Adequate .................................................... 33 2.11 ICM Detailed Activity Descriptions ................................................................. 33 2.12 Focus of Each Commitment Review ................................................................. 36 2.13 Lean Risk Management Plan ........................................................................... 36 2.14 FED Development Process Framework ............................................................ 37 2.15 Steps for Developing Feasibility Evidence ........................................................ 38 2.16 Large-Scale Simulation and Testbed FED Preparation Example ...................... 40 2.17 Example of FED Risk Evaluation Criteria ....................................................... 41 2.18 Conclusions ..................................................................................................... 41 2.19 References....................................................................................................... 41 3 Chapter 3: ICM Fundamentals – Incremental Adoptability.......................................... 43 3.1 Introduction .................................................................................................... 43 3.2 Supplementing Traditional Requirements and Design Reviews with Development and Review of Feasibility Evidence............................................................................ 43 3.3 Stabilized Incremental Development and Concurrent Architecture Rebaselining 43 3.4 Using Schedule as Independent Variable and Prioritizing Features to be Delivered .................................................................................................................. 43 3.5 106753919 Continuous Verification and Validation ........................................................... 43 iv Version Date: 12/31/08 Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects 3.6 Version 0.5 Using the Process Decision Table ..................................................................... 43 4 Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase ............................................................................................................................. 44 4.1 Introduction .................................................................................................... 44 4.2 Phase Key Elements ........................................................................................ 46 4.3 Goals and Objectives ....................................................................................... 48 4.4 Entry Conditions ............................................................................................. 48 4.5 Inputs ............................................................................................................. 48 4.6 Steps (Concurrent) .......................................................................................... 48 4.7 Exit Conditions ............................................................................................... 49 4.8 Work Products ................................................................................................ 49 4.9 Pitfalls ............................................................................................................ 50 5 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table ............. 51 5.1 Introduction .................................................................................................... 51 5.2 The ICM Process Decision Table ..................................................................... 51 5.3 Using the Process Decision Table ..................................................................... 55 5.4 References....................................................................................................... 61 6 Chapter 6: Stage I: Incremental Definition – The Materiel Solution Analysis & AoA/Valuation Phase ..................................................................................................... 62 6.1 Introduction .................................................................................................... 62 6.2 Phase Key Elements ........................................................................................ 62 6.3 Goals and Objectives ....................................................................................... 64 6.4 Entry Conditions ............................................................................................. 64 6.5 Inputs ............................................................................................................. 64 6.6 Steps (Concurrent) .......................................................................................... 64 6.7 Exit Conditions ............................................................................................... 65 6.8 Work Products ................................................................................................ 65 6.9 Pitfalls ............................................................................................................ 65 7 Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase ................................................................................................ 67 7.1 Introduction .................................................................................................... 67 7.2 Phase Key Elements ........................................................................................ 69 7.3 Goals and Objectives ....................................................................................... 71 106753919 v Version Date: 12/31/08 Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects Version 0.5 7.4 Entry Conditions ............................................................................................. 71 7.5 Inputs ............................................................................................................. 71 7.6 Steps (Concurrent) .......................................................................................... 71 7.7 Exit Conditions ............................................................................................... 72 7.8 Work Products ................................................................................................ 72 7.9 Pitfalls ............................................................................................................ 73 8 Chapter 8: Stage II: Incremental Developments – Engineering and Manufacturing Development and Beyond ............................................................................................... 74 8.1 Introduction .................................................................................................... 74 8.2 Phase Key Elements ........................................................................................ 78 8.3 Goals and Objectives ....................................................................................... 80 8.4 Entry Conditions ............................................................................................. 80 8.5 Inputs ............................................................................................................. 80 8.6 Steps (Concurrent) .......................................................................................... 80 8.7 Exit Conditions ............................................................................................... 81 8.8 Work Products ................................................................................................ 81 8.9 Pitfalls ............................................................................................................ 82 8.10 Special Cases ................................................................................................... 82 106753919 vi Version Date: 12/31/08 Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects Version 0.5 Table of Tables Table 0-1. Number of Top-5 Projects Explicitly Using ICM Principles....................................... 17 Table 2-1. Steps for Developing FED ........................................................................................... 39 Table 4-1. Needs and Opportunities (Exploration) ...................................................................... 47 Table 5-1. Characteristics of the Risk-Driven Special Cases of the ICM .................................... 53 Table 5-2. Key Activities for Each Special Case of the ICM ........................................................ 54 Table 6-1 Materiel Solution Analysis; Analysis of Alternatives (Valuation) ................................ 63 Table 7-1. Technology Development and Capability Development Document: TD-CDD (Foundations)................................................................................................................................ 70 Table 8-1. Engineering and Manufacturing Developmentn (Developmentn) ................................ 79 106753919 vii Version Date: 12/31/08 Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects Version 0.5 Table of Figures Figure 0-1. The Defense Acquisition Management System as in DODI 5000.02........................... 6 Figure 0-2. Overview of the Generic Incremental Commitment Life Cycle Process. .................... 7 Figure 0-3. DoD Version of ICM for Evolutionary Deveolpment ................................................ 12 Figure 0-4. Requirements and Acquisition Flow as in DoDI 5000.02 ......................................... 13 Figure 0-5. ICM Activity Categories and Level of Effort ............................................................. 14 Figure 0-6. Feasibility Evidence Description Content ................................................................. 15 Figure 0-7. An Example Combining Directed SoS Engineering and Component Supplier Processes using ICM Anchor Point Reviews. ............................................................................... 16 Figure 2-1. Problems Encountered without FED: 15-Month Architecture Rework Delay .......... 28 Figure 2-2. AT&T Experience with AP Reviews .......................................................................... 30 Figure 2-3. Critical off-Nominal Condition that Cannot Be Handled by the Nominal-Case Architecture................................................................................................................................... 32 Figure 2-4. ICM Detailed Activity Descriptions........................................................................... 35 Figure 2-6. Large-Scale Simulation and Testbed FED Preparation Example ............................ 40 Figure 4-1. Overview of the ICM - Exploration Highlighted ....................................................... 45 Figure 4-2. ICM Activity Categories and Level of Effort ............................................................. 46 Figure 6-1. Overview of the ICM – MSA and AoA Highlighted ................................................... 62 Figure 7-1. Overview of the ICM – Technology Development and CDD Highlighted ................ 68 Figure 7-2. Categorization of DoD Program Assessment Negative Findings ............................. 69 Figure 8-1. Overview of the Generic Incremental Commitment Life Cycle Process – Stage II Highlighted ................................................................................................................................... 74 Figure 8-2. ICM Stage II: Increment Activities ............................................................................ 76 Figure 8-3. DoD Version of ICM for Evolutionary Development ................................................ 78 Figure 8-4. Example of SoSE Coordination Challenge ................................................................ 82 106753919 viii Version Date: 12/31/08 Executive Summary Version 0.5 0Executive Summary 0.1 Abstract Future mission-critical DoD combat platforms, systems of systems, and network-centric services will have many usage uncertainties and emergent characteristics. Their hardware, software, and human factors will need to be concurrently engineered, risk-managed, and evolutionarily developed to converge on cost-effective system operations and mission success. The recent revision of DoDI 5000.02 provides a much improved policy for dealing with these future challenges, including its starting with Needs and Opportunities, its emphasis on early Analysis of Alternatives as an entry condition for Milestone A, its inclusion of a passed Preliminary Design Review as an entry condition for Milestone B, and its emphasis on evolutionary versus single-increment or prespecified increments of development. These emphases on DoDI 5000.02 are also key emphases in the Incremental Commitment Model (ICM), along with more detailed guidance on how to apply them. The ICM has been developed based on best commercial practices for addressing such future challenges; on discussions with contributors to DoDI 5000.02; on successful use of the ICM principles on previous DoD projects; and on its exploratory use on highly complex, future-precursor, net-centric DoD systems of systems. The ICM builds on the experience-based critical success factor principles of stakeholder satisficing, incremental definition, iterative evolutionary system growth, concurrent engineering, and evidence-based, risk-driven management milestones. It is not a one-size-fits-all process model, but has risk-driven options at each milestone decision point to enable projects to adapt to their particular situations. It provides a decision table for early determination of common specialcase life cycle processes to avoid overkill in project ceremony and documentation. It also incorporates the strengths of existing V, concurrent engineering, spiral, agile, and lean process models. As with these other models, it is not fully tested for its ability to support DoDI 5000.02 in all project situations, but it has been formulated to make it straightforward to do so. 0.2 Purpose and Scope The purpose of this Guide is to enable Department of Defense (DoD) projects to better cope with their current and emerging systems engineering challenges, in the context of the recently-revised DoD Instruction 5000.02, “Operation of the Defense Acquisition System.” The definition of the Incremental Commitment Model (ICM) has been co-evolving with the redefinition of the previous DoDI 5000.2. This began with a series of work sessions in 2005 with OUSD (AT&L) personnel to clarify the nature of “spiral development” as recommended in the previous DoDI 5000.2. The redefinitions leading to the ICM were focused both on avoiding common misinterpretations of “Spiral development” and on stimulating systems engineering and acquisition practices better tuned to DoD systems of the future rather than to systems of the past. DoD projects are increasingly encountering “tipping points” in the way their new systems are acquired, the way that existing systems are modified and integrated to form systems of systems (SoSs), and the way that systems that are part of SoSs are modified to contribute to new SoS 106753919 -1- Version Date: 12/31/08 Executive Summary Version 0.5 capabilities. Traditionally, a DoD system’s requirements could be specified in advance, and used as a basis for an often fixed-price, build-to-specification contract to develop given numbers of the systems to be delivered. Several trends have made this approach increasingly obsolete, but the use of traditional DoD systems acquisition and engineering guidance has still been the path of least resistance for DoD program managers. This has become a major root cause for the shortfalls and overruns on recent DoD projects. Recent DoD initiatives besides the revision of DoD 5000.02 such as the Competitive Prototyping initiative and the DoD Systems of Systems Engineering Guide, are beginning to provide the next generation of guidance for DoD systems acquisition. This guidebook is intended to be part of this next generation of guidance for systems engineering of DoD projects. The Incremental Commitment Model (ICM) presented in the Guide is a synthesis of best commercial systems engineering practices and experiences with large DoD space mission systems and precursor SoSs such as the Army’s Future Combat Systems program. The ICM’s overall structure and underlying principles were largely developed as part of a DoD-sponsored National Research Council (NRC) study on Human–System Integration in the System Development Process. The NRC study’s process element involved analyzing the strengths and shortfalls of current systems engineering-related models, and synthesizing a process framework and decision guidelines that capitalized on their strengths while avoiding their shortfalls. The resulting ICM has been used and evolved on systems ranging from small-business and public-service applications to portions of very large DoD SoSs. The full ICM has a number of implications for how future DoD systems would be defined, financed, acquired, and evolved. This Guide identifies these implications, but addressing them is beyond its scope. However, the Guide does show how to selectively apply key ICM practices to strengthen projects being acquired under traditional methods. It also includes a Decision Table that identifies conditions under which traditional DoD systems acquisition and engineering practices remain applicable. The remainder of this Executive Summary provides the motivation and context for using the ICM; a summary of its key distinguishing features and principles; and its relationships to DoDrelevant policies and standards. As DoD policy and guidance becomes more definitive and as improvements over the draft material are incorporated, the Guide will evolve into a definitive version. 0.3 Motivation and Context: Current DoD Needs and Future Challenges Current DoD systems challenges that are driving the need for better systems engineering and acquisition processes include: Complex, multi-owner, net-centric SoS. Although these are complex, they are key to avoiding the shortfalls of current collections of incompatible, separately-developed, stovepiped systems. These shortfalls cause numerous operational deficiencies such as unacceptable delays in service, uncoordinated and conflicting plans, ineffective or dangerous 106753919 -2- Version Date: 12/31/08 Executive Summary Version 0.5 decisions, and inabilities to cope with fast-moving events. Multiple owners of key interdependent systems make synchronization and stabilization of independently evolving systems a major challenge [1]. The ICM’s anchor point milestones have been developed to facilitate such synchronization and stabilization. Emergent requirements. The most appropriate user interfaces and collaboration modes for a complex human-intensive system are not specifiable in advance, but emerge with system prototyping and usage. Forcing them to be prematurely and precisely specified generally leads to poor business or mission performance and expensive late rework and delays. The ICM provides support for incremental and concurrent definition of system requirements and solutions, including competitive prototyping approaches [2]. Rapid change. Specifying current-point-in-time snapshot requirements on a cost-competitive contract generally leads to a big design up front, and a point-solution architecture that is hard to adapt to new developments. Each of the many subsequent changes then leads to considerable nonproductive rework of documents and software (or worse, of hardware), and in renegotiating contracts [3]. The ICM’s incremental definition and development stages directly support the recommendations in [3] for shorter increments and evolutionary development. Reused components and legacy system constraints. Much of current DoD systems engineering and acquisition guidance is oriented around Greenfield development of new systems. However, these systems are increasingly challenged by Brownfield legacy system constraints [19]. Building all of one’s own components from scratch is economically infeasible for complex systems. And at the same time, reuse-based development has major bottom-up development implications that can be incompatible with pure top-down requirements-first approaches. Prematurely specifying requirements (as one project example, hasty specification of a 1-second response time requirement when later prototyping showed that 4 seconds would be acceptable) that disqualify otherwise most cost-effective reusable components often leads to overly expensive, late, and unsatisfactory systems [4]. The ICM includes guidance for reuse-driven and legacy-driven system definition and development. High assurance of qualities. Today’s systems need higher assurance levels of such qualities as safety, security, reliability/availability/maintainability, performance, adaptability, interoperability, usability, and scalability. Just assuring one of these qualities for a complex SoS is difficult. Given the need to “satisfice” (not everybody gets everything they want, but everybody gets something they are satisfied with) among multiple system owners with different quality priorities, complex sources of conflict and tradeoff relationships make multi-attribute satisficing even more challenging [5]. The ICM’s principles of stakeholder satisficing and evidence-based commitment milestones help ensure that key stakeholders’ primary quality concerns are addressed. Increasing importance of software and human factors. Rapid change and emergent requirements imply the need to have rapidly adaptable systems, increasing the importance of software’s flexibility and electronic upgradeability. Increasing SoS complexity presents significant challenges to warfighters’ and commanders’ situation-understanding capabilities, increasing the importance of human engineering. Such concerns led to one of the top recommendations from the October 2006 OUSD (AT&L) Defense Software Strategy 106753919 -3- Version Date: 12/31/08 Executive Summary Version 0.5 Summit: to find ways of better integrating software engineering into the systems engineering and acquisition process [6]. Concurrently, the NRC study was addressing the problem of better integrating human factors into the systems engineering and acquisition process [7]. One of its results was to refine and recommend the ICM and its principles as a process framework for integrating hardware, software, and human factors engineering into the systems engineering process. 0.4 The 6 Key Systems Engineering Process Principles The NRC human-systems integration study analyzed the strengths and difficulties of current process models in addressing these challenges. Each had strengths, but needed further refinements to address all of the six challenges above. The most important conclusion, though, was that there were key process principles that address the challenges, and that forms of the models were less important than their ability to adopt the principles. These key principles are: 1. Commitment and accountability of system engineers, developers, and managers. Without key personnel commitment and accountability for the system under development, there is no way to build trust among the system’s stakeholders. It is too easy to overpromise and depart. As recommended in the NRC Kaminski report [3], development increments must be short enough that the key program leaders remain engaged in their current jobs. And there must be clear visibility of progress versus plans up and down the supplier chain. 2. Stakeholder satisficing. If a system development process presents a success-critical operational or development stakeholder with the prospect of an unsatisfactory outcome, the stakeholder will generally refuse to cooperate, resulting in an unsuccessful system. Stakeholder satisficing includes identifying the success-critical stakeholders and their value propositions; negotiating a mutually satisfactory set of system requirements, solutions, and plans; and managing proposed changes to preserve a mutually satisfactory outcome. 3. Incremental and evolutionary growth of system definition and stakeholder commitment. This characteristic captures the often incremental discovery of emergent requirements and solutions via methods such as prototyping, operational exercises, and use of early system capabilities. Requirements and commitment cannot be monolithic or fully pre-specifiable for complex, human-intensive systems; increasingly detailed understanding, trust, system definition, and commitment is achieved through an evolutionary process. 4. Iterative system development and definition. The incremental and evolutionary approaches lead to cyclic refinements of requirements, solutions, and development plans. Such iteration helps projects to learn early and efficiently about operational and quality requirements and priorities. 5. Concurrent system definition and development. Initially, this includes concurrent engineering of requirements and solutions, and integrated product and process definition. In later ICM increments, change-driven rework and rebaselining of next-increment requirements, solutions, and plans occurs concurrently with stabilized development of the current system increment. This allows early fielding of core capabilities, continual adaptation to change, and timely growth of complex systems without waiting for every requirement and subsystem to be defined. 106753919 -4- Version Date: 12/31/08 Executive Summary Version 0.5 6. Evidence-based, risk driven decision milestones. The key to synchronizing and stabilizing all of this concurrent activity is a set of evidence-based, risk-driven stakeholder decision milestones. At these milestones, evidence of the business, technical, and operational feasibility of the growing package of specifications and plans is evaluated by independent experts. Shortfalls in feasibility evidence are treated as risks and addressed by risk management plans. If the system’s success-critical stakeholders find the risks acceptable and the risk management plans sound, the project will proceed to the next phase. If not, the project can extend its current phase, de-scope its objectives, or avoid low-return resource commitments by terminating the project. If the risks of proceeding straight into development are negligible, the project can skip one or more early phases. Emphasis on feasibility evidence deserves further elaboration. A significant step forward in DoD systems engineering and acquisition guidance was to progress from schedule-based major project reviews (the Preliminary Design review is scheduled for September 30, so we’ll have it then, ready or not) to event-based reviews (the PDR will be held when there is a preliminary design to review). This is better, but frequently leads to “Death by PowerPoint and UML” reviews. These present much detail, but there is little time to determine whether or not the design will meet the system’s key performance parameters. Such evidence of feasibility is generally desired, but not considered as a project deliverable. Thus, it is often neglected when budgets are tight and contractual progress payments and award fees are based on producing a design. In an ICM evidence-based review, the feasibility evidence is a first-class deliverable. As such, its planning and preparation becomes subject to earned value management and factored into progress payments and award fees. Investments in feasibility evidence have been found to pay off significantly in development rework avoidance. In a regression analysis of 161software projects, rework due to shortfalls in architecture and risk resolution grew from 18% added effort for small (10,000 source lines of code) projects, to 91% added effort for very large (10 million SLOC) projects [20]. 0.5 Overview of the Generic ICM and DoD Life Cycle Processes This section presents an overview of the generic ICM life cycle process, with corresponding DoDI 5000.02 phases and milestones shown along with their generic ICM counterparts. Figure 0-1 reproduces DODI 5000.02’s Figure 1 [8] and Figure 0-2 shows its relation to the generic ICM phases. As DoDI 5000.02 focuses on acquisition and not operations, the ICM extends Figure 1 to show reviews for commitments to operations. Section 0.6 will show a more specific DoD version of the ICM based on Figure 2 of DoDI 5000.02. The ICM builds on the strengths of current process models: early verification and validation concepts in the V-model, concurrency concepts in the Concurrent Engineering model, lighterweight concepts in the Agile and Lean models, risk-driven concepts in the spiral model, the phases and anchor points in the Rational Unified Process (RUP) [9, 10], and recent extensions of the spiral model to address SoS acquisition [11]. In comparison to the software-intensive RUP (the closest widely-used predecessor to the ICM), the ICM also addresses hardware and human factors integration. It has extended the RUP phases to cover the full system life cycle: an Exploration phase focused on Needs and Opportunities (as in DoDI 5000.02) precedes the RUP Inception phase, which is refocused on valuation and 106753919 -5- Version Date: 12/31/08 Executive Summary Version 0.5 investment analysis (and for DoD, on Materiel Solution Analysis and Analysis of Alternatives). Following a successful DoD Milestone A review based on evidence of system feasibility, the ICM counterpart of the RUP Elaboration phase and DoD Technology Development and Capability Development Document phase is focused on Foundations. This is a term based on [12] describing concurrent development of requirements, architecture, plans, and feasibility evidence as the essential foundations for the DoD commitment at Milestone B to proceed into Engineering and Manufacturing Development. The RUP Construction and Transition phases are combined into Engineering and Manufacturing Development; and an additional ICM Operations&Production phase combines operations, production, maintenance, and phase-out. The names of the ICM milestones are changed to emphasize that their objectives are to ensure stakeholder commitment to proceed to the next level of resource expenditure based on a thorough feasibility and risk analysis, and not just on the existence of a set of system objectives and a set of architecture diagrams. Thus, the RUP Life Cycle Objectives (LCO) milestone and DoD Milestone A is called the Foundations Commitment Review (FCR) in the ICM, and the RUP Life Cycle Architecture (LCA) milestone and DoD Milestone B is called the Development Commitment Review (DCR). Figure 0-1. The Defense Acquisition Management System as in DODI 5000.02 106753919 -6- Version Date: 12/31/08 Executive Summary Version 0.5 Figure 0-2. Overview of the Generic Incremental Commitment Life Cycle Process. (Note: MDD, A, B, and C are the DoDI 5000.02 acquisition milestones.) 106753919 -7- Version Date: 12/31/08 Executive Summary Version 0.5 In comparison to the sequential waterfall [13] and V-model [14], the ICM explicitly: Begins with exploring Needs and Opportunities, as in DoDI 5000.02, rather than with documenting requirements. Emphasizes concurrent engineering of requirements and solutions Establishes Feasibility Evidence Descriptions as project deliverables and milestone criteria Enables risk-driven avoidance of unnecessary documents, phases, and reviews Provides support for a stabilized current-increment development performed concurrently with a separate change processing and rebaselining activity to prepare for appropriate and stabilized development of the next increment. These aspects can be integrated into a waterfall or V-model, enabling projects required to use such models to cope more effectively with current and future challenges. The overall lifecycle process divides naturally into two major stages. Stage I, Incremental Definition, covers the up-front growth in system understanding, definition, feasibility assurance, and stakeholder commitment leading to a larger Stage II commitment to implement a feasible set of specifications and plans for Incremental Development and Operations of a materiel solution. Stage I: The duration of the generic Stage I can be anywhere from one week to five years. The duration depends on such factors as the number, capability, and compatibility of the proposed system's components and stakeholders. A small, well-jelled agile-methods, developer-customer team operating on a mature infrastructure can form and begin incremental development using Scrum, eXtreme Programming (XP), Crystal or other agile methods in a week. An ultra-large, unprecedented, multi-mission, multi-owner SoS project may take up to five years to progress from a system vision through sorting out needs, opportunities, and organizational roles; maturing key technologies; reconciling infrastructure incompatibilities; reducing uncertainties and risks via models, simulations, prototypes, advanced technology demonstrations, and operational exercises; and evolving a feasibility-validated set of specifications and plans for Stage II. These specifications and plans would be at the build-to level for the initial increment, but only elaborated into detail for the later increments and the overall system where there were high-risk elements to resolve and likely changes in mission objectives, technology, and interoperating systems to accommodate. As shown in Figure 0-2, each project's activity trajectory will be determined by the arrows chosen based on the risk assessments and stakeholder commitment decisions at its anchor point milestone reviews. The small agile project will follow the negligible-risk arrows at the bottom of Figure 0-2 to skip the Valuation and Foundations phases and begin Stage II after a short exploratory phase confirms that the risks of doing so are indeed negligible. The ultra-large project could, for example, apply a form of competitive prototyping to fund four small competitive concept-definition and validation contracts in the Exploratory phase, three larger follow-on Valuation contracts, and two considerably larger Foundations contracts, choosing at each anchor point milestone the best-qualified teams to proceed, based on the feasibility and risk evaluations performed at each anchor point milestone review. Or, in some cases, the reviews might indicate that certain essential technologies or infrastructure incompatibilities needed more work before proceeding into the next phase. 106753919 -8- Version Date: 12/31/08 Executive Summary Version 0.5 Stage II: For Stage II, Incremental Development and Operations, a key decision that is made at the Development Commitment Review is the length of the increments to be used in the system's development and evolution. A small agile project can use two-to-four week increments. However, an ultra-large SoS project with a couple of dozen system suppliers, each with a halfdozen levels of subcontractors, would need increments of up to two years to develop and integrate an increment of operational capability, although with several internal integration subincrements. Some of the non-subsettable hardware systems would take even longer to develop their initial increments, and would be scheduled to synchronize their deliveries with later infrastructure or software increments. The features in each Stage II increment would be prioritized and the increment architected to enable what has variously been called timeboxing, time-certain development, or Schedule As Independent Variable (SAIV), in which borderline-priority features are added or dropped to keep the increment on schedule. It would also be architected to accommodate foreseeable changes, such as user interfaces or transaction formats. For highly mission-critical systems, it would include a continuous verification and validation team analyzing, reviewing, and testing the evolving product to minimize delayed-defect-finding rework. While the stabilized development team is building the current increment and accommodating foreseeable changes, a separate system engineering team is dealing with sources of unforeseeable change and rebaselining the later increments' specifications and plans. Such changes can include new COTS releases, previous-increment usage feedback, current-increment deferrals to the next increment, new technology opportunities, or changes in mission priorities. Having the development team try to accommodate these changes does not work, as it destabilizes their schedules and carefully-worked-out interface specifications. At the end of each increment, the system engineering team also produces for expert review the feasibility evidence necessary to ensure low-risk, stabilized development of the next increment by the build-to-spec team. 0.6 ICM Metaphor and Competitive Prototyping Example A simple metaphor to help understand the ICM is to compare ICM to incremental-commitment gambling games such as poker or blackjack, rather than to single-commitment gambling games such as Roulette. Many system development contracts operate like Roulette, in which a full set of requirements is specified up front, the full set of resources is committed to an essentially fixed-price contract, and one waits to see if the bet was a good one or not. With the ICM, one places an increasing series of bets to see whether the prospects of a win are good or not, and decides whether or not to continue betting based on better information about the prospects of success. An example to illustrate ICM incremental-commitment benefits is the Unmanned Aerial Vehicle (UAV) (or Remotely Piloted Vehicles (RPV)) system enhancement discussed in Chapter 5 of the NRC Human-System Integration report [7]. The RPVs are airplanes or helicopters operated remotely by humans. These systems are designed to keep humans out of harm’s way. However, current RPV systems are human-intensive, often requiring two people to operate a single vehicle. The increase in need to operate numerous RPVs is causing a strong desire to modify the 1:2 (one vehicle controlled by 2 people) ratio to allow for a single operator to operate more than one RPV If a proof-of principle agent-based prototype demo shows how a single operator can control 4 106753919 -9- Version Date: 12/31/08 Executive Summary Version 0.5 RPVs (a 4:1 ratio) in performing some relevant RPV tasks with the help of autonomous agent technology, how should one proceed? A total-commitment approach. A total-commitment approach that is representative of several recent DoD acquisition programs would use the demo results to create a proposed program that uses the agent-based technology to develop a 4:1 ratio system that enables a single operator to control 4 RPVs. A number of assumptions are made to sell the program at an optimistic cost of $1 billion and schedule of 40 months. Enthusiasm is such that the program, budget, and schedule are established, and an RFP is sent out to prospective bidders. The winning bidder provided an even more impressive demo of agent technology and a proposal indicating that all of the problems were well understood, that a preliminary design review (PDR) could be held in 120 days, and that the cost would be only $800M. However, at the PDR, the contractor could not show feasible solutions for several key scenarios, such as coping with inconsistent data, having the individual controllers also coordinate with each other, and satisfying various interoperability protocols. Since the schedule was tight and the contractor provided ideas for addressing the problems, the PDR was declared to be passed. However, the problems were individually more complex than expected and jointly coupled in critical ways, and after 40 months and $800K, some components were developed but experiencing integration problems, and even after descoping the performance to a 1:1 operator:RPV ratio, several problems were still unresolved. Eventually, the 1:1 capability was achieved and the system delivered, but with reduced functionality, a cost of $3 billion, and a schedule of 80 months. An incremental-commitment approach. An incremental-commitment approach, using the ICM process and competitive prototyping, would begin by investing $25M in a pre-MDD Exploration phase. This would initially involve determining key performance parameters and scenarios, including inconsistent data, operator collaboration, and interoperability scenarios, and evaluation criteria and processes. It would then select four bidders to develop virtual prototypes addressing the key issues and providing evidence of the level of performance. The program office would then have a set of independent experts evaluate the bidders’ results. Based on the results, it would determine whether the technology was too immature to merit further current investment as an acquisition program, or that there were acceptable levels of system performance, cost, and risk to invest a next level of resources in addressing the problems identified and developing initial prototype physical capabilities. In this case, the prospects for developing a 4:1 capability were clearly unrealistic, but the prospects for a 1:1 capability were sufficiently attractive to merit another level of investment, corresponding to a Material Solution Analysis - Validation phase. This phase was funded at $75 million, some of the more ambitious key performance parameters were scaled back, and the competitors were downselected to three. The evaluation of the resulting prototypes served as an Analysis of Alternatives, and the top two competitors provided sufficient evidence of feasibility that a Milestone A review was passed, and $225 million was provided for a Technology Development – Foundations phase. In this phase, the competitors not only developed operational prototypes and provided evidence of their ability to satisfy the key performance parameters and scenarios. They also developed Capability Development Documents defining the foundations for a successful Engineering and Manufacturing Development (EMD) phase. These included the ICM Development Commitment 106753919 - 10 - Version Date: 12/31/08 Executive Summary Version 0.5 Review contents of the proposed system’s concept of operation, requirements, architecture, and plans, along with a Feasibility Evidence Description providing evidence that a system built to the architecture would satisfy the requirements and concept of operation, and be buildable within the budgets and schedules in the plan. The feasibility evidence included a few shortfalls, such as remaining uncertainties in the interface protocols with some interoperating systems, but each of these was covered by a risk mitigation plan in the winning competitor’s submission. The resulting Milestone B review was passed, and the winner’s proposed $675 million, 18-month, three-increment EMD-phase plan to develop an Initial Operational Capability was adopted. The resulting IOC was delivered on budget and schedule, with a few lower-priority features deferred to later system increments. While this is a hypothetical case, it shows how a premature total commitment without significant modeling, analysis, and feasibility assessment will often lead to large overruns in costs and schedule, and a performance that is considerably less than initially desired. However, by “buying information” early, the incremental commitment and competitive prototyping approach was able to develop a system with much less late rework than the total commitment approach, and with much more visibility and control over the process. The competitive prototyping approach spent about $150-160 million in unused prototypes, but the overall expenditure was only $1 billion as compared to $3 billion for the total-commitment approach, and the capability was delivered in 40 versus 80 months, which indicates a strong return on investment. Further, the funding organizations had realistic expectations of the outcome, so that a 1:1 capability was a successful realization of an expected outcome, rather than a disappointing shortfall from a promised 4:1 capability. A well-documented real example of the approach is the USAF-TRW CCPDS-R project described in Appendix D of Walker Royce’s book, Software Project Management [21]. 0.7 The DoD Version of the ICM Another view of the DoD version of the ICM used in this Guide is shown in Figure 0-3. It has a few differences from the corresponding Figure 2 in DoDI 5000.02, provided here for comparison as Figure 0-4. Each fits certain situations and is a reasonable approximation for the others. For example, the ICM figure places the beginning of systems engineering preparation for the next increment of Engineering and Manufacturing Development (EMD) at the beginning of the current EMD increment, while the DoDI 5000.02 version shows a gap in systems engineering activity for the next increment. This will be the case if the content of the next increment requires not-yet mature technology. However, there will also be situations in which, as recommended in the NRC Pre-Milestone A and Early-Phase Systems Engineering report [3], there is mature technology available for several future increments, and evolutionary development is used to reduce the risk of systems obsolescence in an era of rapid change. In such situations, the major risks are due to effects of rapid changes in the threat, available technology, mission priorities, interoperating systems, or previous-increment usage feedback. There, it is important to devote a continuous critical mass of systems engineering talent to rebaseline the Capabilities Development Document for the next increment. The ICM’s risk arrows accommodate both situations. 106753919 - 11 - Version Date: 12/31/08 Executive Summary Version 0.5 While the scope of DoDI 5000.02 is focused on the operation of the Defense acquisition system, the ICM scope also includes the pre-acquisition activities leading up to a Materiel Development Decision (MDD), and the post-acquisition activities involving the system’s operational use. Thus, the ICM has an Exploration Commitment Review that covers stakeholder commitment to prepare for an MDD (called an MDP review in Figure 0-2), and an Operations Commitment Review (OCR) for each increment that is different from the DoDI 5000.02 Milestone C (also called an ICM Production Commitment Review), which covers stakeholder commitment to production of the next technology increment. Figure 0-3 shows how the DoD version of the ICM accommodates these differences. As usual, a single diagram cannot cover all the possible sequences of operations and production decisions, but these can be covered for a particular system by separate numbering schemes for operational and production increments. Figure 0-3. DoD Version of ICM for Evolutionary Deveolpment 106753919 - 12 - Version Date: 12/31/08 Executive Summary Version 0.5 Figure 0-4. Requirements and Acquisition Flow as in DoDI 5000.02 0.8 What is Being Concurrently Engineered in the ICM? Having addressed the stages, phases, and milestones in the generic ICM and its DoD version, let us now look at the activities. The top row of Activities in Figure 0-2 indicates that a number of system aspects are being concurrently engineered at an increasing level of understanding, definition, and development. The most significant of these aspects are shown in Figure 0-5, an extension of a similar view of concurrently engineered software projects developed as part of the RUP [9]. As discussed above, the ICM extensions to begin with Envisioning Opportunities and Understanding Needs precisely correspond to the initial activities discussed in Section 3 of DoDI 5000.02 Enclosure 2 [8]. 106753919 - 13 - Version Date: 12/31/08 Executive Summary Version 0.5 Figure 0-5. ICM Activity Categories and Level of Effort As with the RUP version, it should be emphasized that the magnitude and shape of the levels of effort will be risk-driven and likely to vary from project to project. In particular, they are likely to have mini risk/opportunity-driven peaks and valleys, rather than the smooth curves shown for simplicity in Figure 0-5. The main intent of this view is to emphasize the necessary concurrency of the primary success-critical activities shown as rows in Figure 0-5. Thus, in interpreting the Exploration column, although system scoping is the primary objective of the Exploration phase, doing it well involves a considerable amount of activity in understanding needs, envisioning 106753919 - 14 - Version Date: 12/31/08 Executive Summary Version 0.5 opportunities, identifying and reconciling stakeholder goals and objectives, architecting solutions, life cycle planning, evaluating alternatives, and negotiating stakeholder commitments. For example, if one were exploring the initial scoping of the RPV example discussed above, one would not just interview a number of stakeholders and compile a list of their expressed mission needs. One would also envision and explore opportunities for using alternative technologies, perhaps via competitive prototyping. In the area of understanding needs, one would determine relevant key performance parameters, scenarios, and evaluation criteria for evaluating the prototypers’ results. And via the prototypes, one would explore alternative architectural concepts for developing and evolving the system; evaluate their relative feasibility, benefits, and risks for stakeholders to review; and if the risks and rewards are acceptable to the stakeholders, to negotiate commitments of further resources to proceed into a Valuation phase with a clearer understanding of what level of requirements would be feasible. 0.9 How is All This Concurrent Engineering Synchronized and Stabilized? Figure 0-5 indicates that a great deal of concurrent activity occurs within and across the various ICM phases. To make this concurrency work, the evidence-based anchor point milestone reviews are the mechanisms by which the many concurrent activities are synchronized, stabilized, and risk-assessed at the end of each phase. Each of these anchor point milestone reviews, labeled at the top of Figure 0-5, is focused on developer-produced and independentexpert reviewed evidence, instead of assertions, assumptions, PowerPoint charts and Unified Modeling Language (UML) diagrams, to help the key stakeholders determine the next level of commitment. The review processes and use of independent experts are based on the highly successful AT&T Architecture Review Board procedures described in [16]. Figure 0-6 shows the content of the Feasibility Evidence Description. Showing feasibility of the concurrently-developed elements helps synchronize and stabilize the concurrent activities. Feasibility Evidence Description Content Evidence provided by developer and validated by independent experts that if the system is built to the specified architecture, it will: – Satisfy the requirements: capability, interfaces, level of service, and evolution – Support the operational concept – Be buildable within the budgets and schedules in the plan – Generate a viable return on investment – Generate satisfactory outcomes for all of the success-critical stakeholders – Resolve all major risks by treating shortfalls in evidence as risks and covering them by risk management plans – Serve as basis for stakeholders’ commitment to proceed Figure 0-6. Feasibility Evidence Description Content 106753919 - 15 - Version Date: 12/31/08 Executive Summary Version 0.5 The Operations Commitment Review (OCR) is different, in that it addresses the often much higher operational risks of fielding an inadequate system. In general, stakeholders will experience a factor of two-to-ten increase in commitment level in going through the sequence of ECR/MDP to DCR/B milestones, but the increase in going from DCR/B to OCR can be much higher, as can be the increase in going from DCR to the Production Commitment Review (PCR or Milestone C in DoDI 5000.02). These commitment levels are based on typical cost profiles across the various stages of the acquisition life-cycle. The OCR focuses on evidence of the adequacy of plans and preparations with respect to doctrine, organization, training, materiel, leadership, personnel, and facilities (DOTMLPF), along with plans, budgets, and schedules for fielding and operations. Taking this to the next level for “directed” System of Systems (SoS) development (the case where there is an SoS engineering team with considerable responsibility and authority to “direct” the development of the SoS components), Figure 0-7 shows how these anchor point milestone reviews can be used to synchronize, stabilize, and manage risks across multiple supplier/vendor/strategic partner activities. Figure 0-7. An Example Combining Directed SoS Engineering and Component Supplier Processes using ICM Anchor Point Reviews. The major SoS-level milestones are compatible with those of Figure 0-2, but the realities of many SoSs involve some reinterpretation of their nature. Many SoSs will need to include COTS, legacy, or separately managed systems that are defined and incrementally released on different schedules than the SoS in Figure 0-7. The case that is shown in Figure 0-7 is one in which for reasons of training, provisioning, or operational stability, the main upgrades are batched into major SoS operational releases. Other SoS cases may require a more continual stream of upgrades such as security patches or electronic warfare countermeasures, in which parts of the SoS are more continuously versus incrementally evolving. Since not all of the source-selected component systems are defined on the same schedule, there will be delays in reconciling them into a common system architectural framework, requiring an additional SoS Rebaseline/Adjustment Foundations Commitment Review after the original SoS FCR/A milestone used to select superior alternatives. The selected suppliers and partners will 106753919 - 16 - Version Date: 12/31/08 Executive Summary Version 0.5 also need to participate in negotiating the particular SoS build-to architecture that will be baselined at the SoS DCR/B milestone. And their incremental delivery schedules will not all be compatible with the incremental delivery schedule of the SoS in Figure 0-7. Some of these suppliers, such as Supplier x, will already be operating and will feed incremental upgrade information into the SoS process during its definition and development stages. The SoS system engineers will try to predefine and anticipate these upgrades as much as possible, but will have to adapt to changes in Supplier x's capabilities or interfaces. If these are well-anticipated, the SoS development team in Figure 0-2 will accommodate them. If they are complex and unanticipated, threatening destabilization of the build-to-spec team, the SoS architecting rebaselining team will engineer an interim solution if necessary to keep the operational capability running. If a more comprehensive solution is needed for the longer term, the team will rebaseline the SoS architecture of the next increment to accommodate Supplier x's new capability. Changes from Strategic Partner C and Supplier B would be handled similarly. In some additional cases, particularly for Supplier A who may be developing a longer-duration, non-subsettable, hardware-intensive initial operational capability, the SoS management will schedule Supplier A's IOC to synchronize with a later SoS increment (OCR2 in Figure 0-7). Even more complex are the “Acknowledged” and “Collaborative” SoS forms, which contain multiple system owners and management chains to deal with. Chapter 14 discusses how the ICM can be tailored to address these increasingly important SoS forms. 0.10 Project Experience with ICM Principles One of the origins of the ICM was an effort during its formulation to capture commercial best practices for incorporating human-system factors into the system life cycle process in the NRC study on that topic [7]. A good example of commercial use of these best practices in an ICM process framework is an award-winning, commercially-successful medical infusion pump development involving complex integration of leading-edge hardware, software, and human factors technology for a safety-critical product. It is documented in Chapter 5 of [7]. Another good source of successful projects that have applied ICM Critical Success Factor principles is the annual series of Top-5 software-intensive systems projects published in CrossTalk [17]. The “Top-5 Quality Software Projects” were chosen annually by panels of leading experts as role models of best practices and successful Year Concurrent RiskEvolutionary Engineering Driven Growth outcomes. Table 0-1 summarizes each year’s record with respect to usage of four of the six 2002 4 3 3 principles: concurrent engineering; risk-driven 2003 5 4 3 activities; and evolutionary, iterative system 2004 3 3 4 growth (most of the projects were not specific about system sponsor commitment and 2005 4 4 5 accountability or stakeholder satisficing). Of Total (of 20) 16 14 15 the 20 Top-5 projects in 2002 through 2005, 16 explicitly used concurrent engineering; 14 explicitly used risk-driven development; and 15 explicitly used evolutionary, iterative system Table 0-1. Number of Top-5 Projects Explicitly Using ICM Principles 106753919 - 17 - Version Date: 12/31/08 Executive Summary Version 0.5 growth, while additional projects gave indications of their partial use. Evidence of successful results of stakeholder-satisficing can be found in the annual series of University of Southern California (USC) e-Services projects using the Win-Win Spiral model as described in [18]. Since 1998, over 50 user-intensive e-Services applications have used this precursor and subsequent evolving versions of the ICM to achieve a 92% success rate of on-time delivery of stakeholder-satisfactory systems. 0.11 Structure and Content of the Guide The Guide comprises of 21 chapters, not including this Executive Summary. Below is a short overview of each chapter. Section I. ICM Fundamentals Chapter 1: ICM Fundamentals – Fundamental Principles. This chapter deals specifically with the six key principles at the core of the ICM: 1 Commitment and accountability of system engineers, developers, and managers; 2. Stakeholder satisficing; 3. Incremental and evolutionary growth of system definition and stakeholder commitment; 4. Iterative system development and definition; 5. Concurrent system definition and development; and 6. Evidencebased, risk driven decision milestones. It includes more thorough explanations of these principles in the context of the ICM, along with examples of how these were used successfully in some of the CrossTalk “Top-5 Quality Software Projects.” Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs. The DoDI 5000.02 counterparts of the ICM AP (Anchor Point) Milestones and FEDs (Feasibility Evidence Descriptions) are discussed in this chapter. They are the Materiel Decision Preparation/Exploration Commitment Review (MDP/ECR), the Materiel Development Decision/Valuation Commitment Review (MDD/VCR), the Milestone A/Foundations Commitment Review (A/FCR), the Development Commitment Review (Milestone B/DCR), the DoDI 5000.02 Milestone C Production and Deployment commitment review (C/PCR) and the ICM Operations Commitment Review (OCR). This chapter will explain the milestones and FEDs on a fundamental level, e.g. What are examples of FEDs? Why is an FED required at each Anchor Point? How do they help synchronize and stabilize concurrent activities? How do they help in risk management? How is their progress tracked as a basis of earned value through each phase? Chapter 3: ICM Fundamentals – Incremental Adoptability. Current system development programs are already operating under a set of contractual standards that may not fit well with their need to operate as or within a system of systems; to handle emergent requirements; or to cope with rapid change. For such programs, it is often not possible to make major process changes all at once. Rather, it is better to incrementally change processes over time. This chapter provides guidance for incremental adoption of key ICM processes. 106753919 - 18 - Version Date: 12/31/08 Executive Summary Version 0.5 Section II. ICM Stage I: Incremental Definition Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase. The ICM Exploration Phase is focused on system scoping based upon the need for new or upgraded capabilities. This phase explores materiel development and other alternatives by investigating and understanding needs, envisioning opportunities, identifying and reconciling stakeholder goals and objectives, negotiation of stakeholder commitments, and preparing for a Materiel Development Decision by such activities as establishing guidelines for the analysis of alternatives. This chapter describes these activities in more detail. Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table. As with other models trying to address a wide variety of situations, the general form of the ICM is rather complex. However, its risk-driven nature has enabled us to determine a set of twelve common risk patterns and organize them into a decision table that can help projects new to the ICM converge on a process that fits well with their particular process drivers. For each of the twelve special cases, the decision table provides top-level guidelines for tailoring the key activities of the ICM, along with suggested lengths between each internal system build and each external system increment delivery. This chapter elaborates on each of the twelve cases and provides examples of their use. Chapter 6: Stage I: Incremental Definition – The Materiel Solution Analysis & AoA/Valuation Phase. If the results of the MDD Preparation/Exploration Phase indicate that there are system capability needs worth pursing further, the Materiel Development Decision (MDD) is passed positively, and a commitment is made to go on to the next phase. The Materiel Solution Analysis & AoA/Valuation Phase is focused on further system definition, valuation and investment analysis. As part of this effort, an analysis of alternatives (AoA) is conducted using the guidance provided in the MDD Preparation/Exploration Phase. This chapter describes these activities and illustrates them though examples in more detail. Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase. If the results of the ICM Materiel Solution Analysis & AoA Valuation phase indicate that there are viable, cost-effective options for providing the needed capability, a commitment is made to move forward into the Technology Development & CDD/Foundations Phase. For the DoD acquisition process, this Commitment is the same as the Milestone A commitment. The TD&CDD/Foundations Phase focuses not only on technology development but also on the concurrent development of a Capability Development Document (CDD) covering the requirements, architecture, and plans needed to provide a solution. These are the needed “foundations” upon which the system will be developed in Stage II, and whose feasibility evidence will be assessed in making a Milestone B/ Development Commitment Review decision to proceed into Engineering and Manufacturing Development. This chapter describes these activities and illustrates them through examples in more detail. 106753919 - 19 - Version Date: 12/31/08 Executive Summary Version 0.5 Section III. ICM Stage II: Incremental Development Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V. If a viable, cost effective solution is identified in the TD&CDD/Foundations Phase, a commitment is made to proceed with development. For the DoD acquisition process, this commitment is the same as the Milestone B commitment. The Engineering and Manufacturing Development Phase focuses on the development of the system (or deployable increment of a system) employing continual V&V activities to ensure a successful development. This chapter describes and illustrates these activities in more detail. Chapter 9: Stage II: Incremental Development – Concurrent Next-Increment Rebaseline. A key feature of the ICM is that as the current increment is undergoing stable development, a parallel team is handling unanticipated change traffic and rebaselining the foundations for the next increment of development. Note that under the DoDI 5000.02 acquisition process, these activities will generally require additional Milestone A and/or Milestone B Commitments, depending upon the scope of the next planned increment and the maturity of the planned technologies for the next increment. This chapter describes and illustrates these activities in more detail. Section IV. Key ICM Practices and Applications Chapter 10: Key ICM Practices and Applications – Competitive Prototyping. This chapter discusses how the ICM supports competitive prototyping (CP) in the DoD context. It also provides guidance for assessing the costs associated with CP as “buying information to reduce risk”, determining “how much CP is enough?” and for dealing with multiple rounds of CP, as in the RPV CP example above. Chapter 11: Key ICM Practices and Applications – Stakeholder Satisficing. This chapter describes the role of stakeholder satisficing in the successful definition and development of system capabilities for large systems and systems of systems with multiple stakeholders. It provides guidance on identifying success-critical stakeholders, and addressing the most likely conflicts among the various classes of key DoD stakeholders. Examples are provided that show how various trades and negotiations can be conducted. Chapter 12: Key ICM Practices and Applications – Risk Assessment and Management. Realistic risk assessment at each anchor point and active risk management during each phase are key to deciding when to proceed to the next phase, when to investigate alternatives further before proceeding to the next phase, and when to discontinue because the alternatives are too risky. This chapter provides guidance and examples of the role of risk assessment and management at each of the Commitment review points, for determining how much of an activity or artifact is enough, and for such practices as CP. Chapter 13: Key ICM Practices and Applications – Estimation and Management Metrics. As processes evolve to handle current and future challenges, so too must estimation techniques and management metrics. Some of these challenges include emergent requirements, rapid change, net-centric system of systems, model-driven system development, service oriented 106753919 - 20 - Version Date: 12/31/08 Executive Summary Version 0.5 architectures, Brownfield development, always-on never-fail system development, competitive prototyping, hardware/software/systems engineering integration, complex systems, development process changes that impact estimation processes, and shifts in key work products used to communicate information between various system development participants. This chapter explores options and provides best-available guidance for meeting challenges such as these. Chapter 14: Key ICM Practices and Applications – ICM for Systems of Systems. Recent DoD research [22] has found that traditional processes need to be adapted to handle the special challenges presented by the creation and evolution of systems of systems. This research has also identified several types of SoS, with each type characterized by the level of authority and responsibility used to guide the evolution of the SoS. ICM’s flexible and tailorable approach to system development processes is well-suited for these various types of SoSs. This chapter examines various process approaches and examples for the “acknowledged” or “directed” SoS types identified in [22], as well as general support guidance for the “collaborative” SoS type. Chapter 15: Key ICM Practices and Applications – Implications for Acquisition Management and Staffing. The ICM’s emphasis in Stage I on concurrent engineering and on evidence-based versus schedule-based or event-based reviews has several key implications for how to manage and staff the pre-Milestone B activities. These include higher levels of investment but often shorter schedules for systems engineering of complex projects; new management metrics to track as discussed in Chapter 13; and ways of funding Stage I that ensure continuity of effort but also the ability to redirect or discontinue efforts with too-high or unacceptable risk of continuing on their current course. The ICM’s emphasis in Stage II is on concurrent stable development of the current increment; continuous V&V of the current increment; and a parallel team handling unanticipated change traffic and rebaselining the foundations for the next increment of development. This implies the need for different contractual vehicles, incentive structures, and staffing career paths for the three concurrent classes of activity. Section V. Key ICM System Definition Artifacts Chapters 16 through 21: Key System Definition Artifacts. These chapters provide guidance and examples for the key ICM system definition artifacts. These include the operational concept definition, requirements description, architecture description, life cycle plan, systems engineering management plan, and feasibility evidence descriptions at different points in the life cycle. 0.12 Conclusions Future mission-critical DoD combat platforms, systems of systems, and network-centric services will have many usage uncertainties and emergent characteristics. Their hardware, software, and human factors will need to be concurrently engineered, risk-managed, and evolutionarily developed to converge on cost-effective system operations and mission success. The recent revision of DoDI 5000.02 provides a much improved policy for dealing with these future challenges, including its starting with Needs and Opportunities, its emphasis on early 106753919 - 21 - Version Date: 12/31/08 Executive Summary Version 0.5 Analysis of Alternatives as an entry condition for Milestone A, its inclusion of a passed Preliminary Design Review as an entry condition for Milestone B, and its emphasis on evolutionary versus single-increment or prespecified increments of development. These emphases on DoDI 5000.02 are also key emphases in the Incremental Commitment Model (ICM), along with more detailed guidance on how to apply them. The ICM has been developed based on best commercial practices for addressing such future challenges; on discussions with contributors to DoDI 5000.02; on successful use of the ICM principles on previous DoD projects; and on its exploratory use on highly complex, future-precursor, net-centric DoD systems of systems. The ICM builds on the experience-based critical success factor principles of stakeholder satisficing, incremental definition, iterative evolutionary system growth, concurrent engineering, and evidence-based, risk-driven management milestones. It is not a one-size-fits-all process model, but has risk-driven options at each milestone decision point to enable projects to adapt to their particular situations. It provides a decision table for early determination of common specialcase life cycle processes to avoid overkill in project ceremony and documentation. It also incorporates the strengths of existing V, concurrent engineering, spiral, agile, and lean process models. As with these other models, it is not fully tested for its ability to support DoDI 5000.02 in all project situations, but it has been formulated to make it straightforward to do so. 0.13 References 1. Krygiel, A. (1999) Behind the Wizard’s Curtain; CCRP Publication Series, July. 2. Young, J. (2007) Prototyping and Competition, USD(AT&L) Memorandum, 19 September. 3. Kaminski, P. et al.(2008) Pre-Milestone A and Early-Phase Systems Engineering, The National Academies Press. 4. Boehm, B. (2000) “Unifying Software Engineering and Systems Engineering”, Computer, March 2000, pp. 114-116. 5. Clements, P. et al. (2002). Evaluating Software Architectures, Addison Wesley. 6. Baldwin, K. (2007) "DoD Software Engineering and System Assurance: New Organization, New Vision," http://csse.usc.edu/events/2007/ARR/presentations/Baldwin.ppt, February 14. 7. Pew, R. W., and Mavor, A. S., (2007). Human-System Integration in the System Development Process: A New Look, National Academy Press. 8. USD(AT&L) (2008) Operation of the Defense Acquisition System, DoD Instruction 5000.02, December 8. 9. Kruchten, P. (1999). The Rational Unified Process, Addison Wesley. 10. Boehm, B. (July 1996). Anchoring the Software Process. Software. pp. 73-82. 11. Boehm, B., and Lane, J. (May 2006). 21st Century Processes for Acquiring 21st Century Systems of Systems. CrossTalk. pp. 4-9. 106753919 - 22 - Version Date: 12/31/08 Executive Summary Version 0.5 12. Rechtin, E. (1991). Systems Architecting, Prentice Hall. 13. Royce, W. W. (1970). Managing the development of large software systems: concepts and techniques, Proceedings, Wescon 1970. 14. Patterson, F. (1999). System engineering life cycles: life cycles for research, development, test, and evaluation; acquisition; and planning and marketing, in Sage, A., and Rouse, W., ed., (1999) Handbook of Systems Engineering and Management, Wiley, pp. 59-111. 15. U.S Department of Defense (2003). DoD Instruction 5000.2, Operation of the Defense Acquisition System. 16. Maranzano, J. et al. (2005) Architecture reviews: Practice and experience. IEEE Software, March/April 2005, pp. 34-43. 17. CrossTalk (2002-2005), “Top Five Quality Software Projects,” January 2002, July 2003, July 2004, September 2005. 18. Boehm, B., Egyed, A., Kwan, J., Port, D., Shah, A., and Madachy, R. (July 1998). Using the WinWin Spiral Model: A Case Study. Computer. pp.33-44. 19. Hopkins, R., and Jenkins, K. (2008) Eating the IT Elephant: Moving from Greenfield Development to Brownfield, IBM Press. 20. Boehm, B., Valerdi, R., and Honour, E. (2008) The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems, Systems Engineering, Vol. 11, No.3, pp. 221-234. 21. Royce, W. E. (1998) Software Project Management. Addison Wesley. 22. OUSD(AT&L)/Systems and Software Engineering (2008), Systems Engineering Guide for System of Systems, Version 1.0, June. 106753919 - 23 - Version Date: 12/31/08 Chapter 1: ICM Fundamentals – Fundamental Principles Version 0.5 1Chapter 1: ICM Fundamentals – Fundamental Principles 1.1 Introduction The DoD-sponsored National Research Council (NRC) study on Human–System Integration in the System Development Process analyzed the strengths and difficulties of current process models in addressing six challenges: complex, multi-owner, net-centric SoS; emergent requirements; rapid change; reused components and legacy system constraints; high assurance of qualities; and increasing importance of software and human factors. Each had strengths, but needed further refinements to address all of the six challenges. The most important conclusion, though, was that there were key process principles that address the challenges, and that forms of the models were less important than their ability to adopt the principles. 1.2 Key Principle #1: Commitment and Accountability of System Engineers, Developers, and Managers Without key personnel commitment and accountability for the system under development, there is no way to build trust among the system’s stakeholders. It is too easy to overpromise and depart. As recommended in the NRC Kaminski report [1], development increments must be short enough that the key program leaders remain engaged in their current jobs. And there must be clear visibility of progress versus plans up and down the supplier chain. [Further elaboration TBD] 1.3 Key Principle #2: Stakeholder Satisficing If a system development process presents a success-critical operational or development stakeholder with the prospect of an unsatisfactory outcome, the stakeholder will generally refuse to cooperate, resulting in an unsuccessful system. Stakeholder satisficing includes identifying the success-critical stakeholders and their value propositions; negotiating a mutually satisfactory set of system requirements, solutions, and plans; and managing proposed changes to preserve a mutually satisfactory outcome. [Further elaboration TBD] 1.4 Key Principle #3: Incremental and Evolutionary Growth of System Definition and Stakeholder Commitment This characteristic captures the often incremental discovery of emergent requirements and solutions via methods such as prototyping, operational exercises, and use of early system capabilities. Requirements and commitment cannot be monolithic or fully pre-specifiable for complex, human-intensive systems; increasingly detailed understanding, trust, definition and commitment is achieved through an evolutionary process. [Further elaboration TBD] 106753919 - 24 - Version Date: 12/31/08 Chapter 1: ICM Fundamentals – Fundamental Principles 1.5 Version 0.5 Key Principle #4: Iterative System Development and Definition The incremental and evolutionary approaches lead to cyclic refinements of requirements, solutions, and development plans. Such iteration helps projects to learn early and efficiently about operational and quality requirements and priorities. [Further elaboration TBD] 1.6 Key Principle #5: Concurrent System Definition and Development Initially, this includes concurrent engineering of requirements and solutions, and integrated product and process definition. In later increments, change-driven rework and rebaselining of next-increment requirements, solutions, and plans occurs concurrently with stabilized development of the current system increment. This allows early fielding of core capabilities, continual adaptation to change, and timely growth of complex systems without waiting for every requirement and subsystem to be defined. [Further elaboration TBD] 1.7 Key Principle #6: Evidence-Based, Risk Driven Decision Milestones The key to synchronizing and stabilizing all of this concurrent activity is a set of evidence-based, risk-driven stakeholder decision milestones. At these milestones, evidence of the business, technical, and operational feasibility of the growing package of specifications and plans is evaluated by independent experts. Shortfalls in feasibility evidence are treated as risks and addressed by risk management plans. If the system’s success-critical stakeholders find the risks acceptable and the risk management plans sound, the project will proceed to the next phase. If not, the project can extend its current phase, de-scope its objectives, or avoid low-return resource commitments by terminating the project. If the risks of proceeding straight into development are negligible, the project can skip one or more early phases. [Further elaboration TBD] 1.8 References 1. Kaminski, P. et al.(2008) Pre-Milestone A and Early-Phase Systems Engineering, The National Academies Press. 106753919 - 25 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 2Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs 2.1 Developing and Using Feasibility Evidence for Life Cycle Commitment Milestones Success-Critical Stakeholders (SCSHs) for systems under development have the authority and the responsibility to exercise accountable governance of their organization’s resources when committing such resources to the next phase of system development. In many cases, when using processes other than the ICM, these stakeholders are presented with insufficient information upon which to make a responsible decision. As a result, many projects proceed into system development with little idea of whether their project requirements, architectures, plans, budgets, and schedules are feasible and consistent or not. This is one of the major reasons for the relatively low (20-30%) success rates for successful on-time, in-budget system deliveries in the commercial Standish Reports and government GAO reports. The ICM provides guidance for avoiding such system overruns and shortfalls by using APs and FEDs. The main distinction of using APs and FEDs is to provide evidence of feasibility as part of the system definition’s core deliverables, not just assertions, assumptions, or aspirations placed in optional appendices. APs and FEDs emphasize that independent experts should evaluate the evidence supplied by developers. Any shortfalls in evidence should be identified as sources of project risk. These sources of project risk need to be covered by risk mitigation plans. APs and FEDs are an integral part of the ICM discussed in [Boehm-Lane, 2007; Boehm-Lane, 2008; and Pew-Mavor, 2007]. However, they are applicable to models other than the ICM. A good approximation to their use may be obtained by adding FEDs to the content of traditional waterfall or V-model milestones such as System Requirements Reviews (SRRs), System Functional Reviews (SFRs), and Preliminary Design Reviews (PDRs). 2.2 Nature of FEDs and AP Milestones FEDs are evidence provided by the developer and validated by independent experts that, if the system is built to the specified architecture it will: 1. Satisfy the specified operational concept and requirements, including capability, interfaces, level of service, and evolution 2. Be buildable within the budgets and schedules in the plan 3. Generate a viable return on investment 4. Generate satisfactory outcomes for all of the SCSHs 5. Identify shortfalls in evidence as risks, and cover them with risk mitigation plans 106753919 - 26 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 An FED does not assess a single sequentially developed system definition element, but the consistency, compatibility, and feasibility of several concurrently-engineered elements. To make this concurrency work, a set of AP milestone reviews are performed to ensure that the many concurrent activities are synchronized, stabilized, and risk-assessed at the end of each phase. Each of these AP reviews is focused on developer-produced and expert-validated evidence, documented in the FED, to help the key stakeholders—the SCSHs—determine whether to proceed into the next level of commitment. Hence, they are called Commitment Reviews. The FED is based on evidence from simulations, models, or experiments with planned technologies and increasingly detailed analysis of development approaches and project productivity rates. The parameters used in the analyses should be based on measured component performance or on historical data showing relevant past performance, cost estimation accuracy, and actual developer productivity rates. A shortfall in feasibility evidence indicates a level of program execution uncertainty and a source of program risk. It is often not possible to fully resolve all risks at a given point in the development cycle, but known, unresolved risks need to be identified and covered by risk management plans, including the necessary staffing and funding to address them. The nature of the evidence shortfalls, the strength and affordability of the risk management plans, and the stakeholders’ degrees of risk acceptance or avoidance will determine their willingness to commit the necessary resources to proceed. A program with risks is not necessarily bad, particularly if it has strong risk management plans. A program with no risks may be high on achievability, but low on ability to produce a timely competitive advantage. A program more familiar with a sequential waterfall or V-model can achieve most of the effect of a FED-based anchor point milestone review by adding an FED to the set of artifacts to be developed and reviewed for adequacy and satisfaction at a System Requirements Review (SRR), System Functional Review (SFR), or Preliminary Design Review (PDR). In principle, some guidance documents indicate that such feasibility evidence should be produced and reviewed. But since the evidence is not generally specified as a developer deliverable and is vaguely defined, it is generally inadequate as a basis for stakeholder commitments. 2.3 Problems Encountered without FED In the early 1980s, a large government organization contracted with TRW to develop an ambitious information query and analysis system. The system would provide more than 1,000 users, spread across a large building complex, with powerful query and analysis capabilities for a large and dynamic database. TRW and the customer specified the system using a classic sequential-engineering waterfall development model. Based largely on user need surveys, an oversimplified high-level performance analysis, and a short deadline for getting the To-Be-Determineds out of the requirements specification, they fixed into the contract a requirement for a system response time of less than one second. Subsequently, the software architects found that subsecond performance could only be provided via a highly customized design that attempted to anticipate query patterns and cache copies of data so that each user’s likely data would be within one second’s reach (a 1980’s precursor of Google). The resulting hardware architecture had more than 25 super-midicomputers (an earlier 106753919 - 27 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 term for a computer with performance and capacity between a minicomputer and a mainframe) busy caching data according to algorithms whose actual performance defied easy analysis. The scope and complexity of the hardware-software architecture brought the estimated cost of the system to nearly $100 million, driven primarily by the requirement for a one-second response time. Faced with this unattractive prospect (far more than the customer’s budget for the system), the customer and developer decided to develop a prototype of the system’s user interface and representative capabilities to test. The results showed that a four-second response time would satisfy users 90 percent of the time. A four-second response time, with special handling for high-priority transactions, dropped development costs closer to $30 million. Thus, the premature specification of a 1-second response time neglected the risk of creating an overexpensive and time-consuming system development. Fortunately, in this case, the only loss was the wasted effort on the expensive-system architecture and a 15-month delay in delivery (see Figure 2-1). More frequently, such rework is done only after the expensive full system is delivered and found still too slow and too expensive to operate. Figure 2-1. Problems Encountered without FED: 15-Month Architecture Rework Delay 2.4 Problems Avoidable with FED Had the developers been required to deliver a FED showing evidence of feasibility of the onesecond response time, they would have run benchmarks on the best available commercial query systems, using representative user workloads, and would have found that the best that they could do was about a 2.5-second response time, even with some preprocessing to reduce query latency. They would have performed a top-level architecture analysis of custom solutions, and concluded 106753919 - 28 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 that such 1-second solutions were in the $100 million cost range. They would have shared these results with the customer in advance of any key reviews, and found that the customer would prefer to explore the feasibility of a system with a commercially-supportable response time. They would have done user interface prototyping and found much earlier that the four-second response time was acceptable 90% of the time. As some uncertainties still existed about the ability to address the remaining 10% of the queries, the customer and developer would have agreed to avoid repeating the risky specification of a fixed response time requirement. The customer and developer would instead define a range of desirable-to-acceptable response times, with an award fee provided for faster performance. They would also have agreed to reschedule the next milestone review to give the developer time and budget to present evidence of the most feasible solution available, using the savings over the prospect of a $100 million system development as rationale. This would have put the project on a more solid success track over a year before the actual project discovered and rebaselined itself, and without the significant expense that went into the unaffordable architecture definition. For example, the customer and developer would negotiate response time ranges. They would say two-second response time desirable, four-second response time acceptable with some twosecond special cases. Next they would benchmark commercial system add-ons to validate their feasibility. Finally, they would present solution and feasibility evidence at AP milestone review. The result would be an acceptable solution with minimal delay. 2.5 AT&T Experience with AP Reviews AT&T and its spinoffs (Telcordia, Lucent, Avaya, regional Bell companies) have been successfully using versions of AP reviews since 1988, see Figure 2-2. On average, there has been 10% savings per reviewed project, with substantially larger savings on a few reviewed projects. More detail on their Architecture Review experience is in J. Maranzano et. al., “Architecture Reviews: Practice and Experience,” IEEE Software, March/April 2005. 106753919 - 29 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 Figure 2-2. AT&T Experience with AP Reviews USC has used APs and FEDs to achieve a 92% on-time, in-budget, satisfied-client success rate on over 50 fixed-schedule campus e-service projects. AP Reviews are also an integral part of the Rational Software Process, with many successful implementations. 2.6 Overview of ICM Phases, Anchor Points, and Decision Options Figure 0-1 shows how the ICM spans the life cycle process from concept exploration to operations. Each phase includes a number of concurrent activities that are synchronized and stabilized with a FED-based anchor point milestone review at points corresponding with the major DoD acquisition milestones. At each anchor point, there are 4 options, based on the assessed risk of the proposed system. Some options involve go-backs. These options result in many possible process paths. The life cycle is divided into two stages: Stage I of the ICM (Definition) has 3 decision nodes with 4 options/node, culminating with incremental development in Stage II (Development and Operations). Stage II has an additional 2 decision nodes, again with 4 options/node. One can use ICM risk patterns to generate frequently-used processes with confidence that they fit the situation. Initial risk patterns can generally be determined in the Exploration phase. One then proceeds with development as a proposed plan with risk-based evidence at the VCR/Material Development Decision milestone, adjusting in later phases as necessary. Risks associated with the system drive the life cycle process. Information about the risks (feasibility assessments) supports the decision to proceed, adjust scope or priorities, or cancel the program. The downward arrows at the end of each phase provide built-in off-ramps for clearly 106753919 - 30 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 infeasible programs. The horizontal arrows provide additional options to skip phases or activities if the risk of doing so is negligible. 2.7 Key Point: Need to Show Evidence It was a significant step forward for DoD to progress from having schedule-based reviews, where the review took place at the scheduled time whether the project was ready or not, to event-based reviews, where the review would not take place until its artifacts—e.g., functional requirements—were completed. However, many event-based reviews focused on hastily-produced artifacts that had not been validated for compatibility or feasibility, such as the example in Figure 2-1. Thus, it is another significant step forward to progress from event-based reviews to evidence-based reviews. The major difference in the example project proceeding without and with an FED was that the need to provide evidence of feasibility requires the project to develop and exercise benchmarks and prototypes that expose both infeasible solutions and unnecessarily ambitious requirements. FED needs to be more than just traceability matrices and PowerPoint charts. Evidence can include results of: Prototypes: of networks, robots, user interfaces, COTS interoperability Benchmarks: for performance, scalability, accuracy Exercises: for mission performance, interoperability, security Models: for cost, schedule, performance, reliability; tradeoffs Simulations: for mission scalability, performance, reliability Early working versions: of infrastructure, data fusion, legacy compatibility Previous experience Combinations of the above Not only does the evidence need to be produced, but it needs to be validated by independent experts. These experts need to determine the realism of assumptions, the reperesentativeness of scenarios, the thoroughness of analysis, and the coverage of key off-nominal conditions. The risk of validating just nominal-case scenarios is shown in the next section. 2.8 Off-Nominal Architecture-Breakers Often, projects with tight schedules will overfocus their architecture and development effort on the nominal-case requirements. Figure 2-3 shows the frequent result: a critical off-nominal condition that cannot be handled by the nominal-case architecture, causing large amounts of rework and delays, and reproducing the common Pareto diagram in which 20% of the defects cause 80% of the work. This also happens on programs that suboptimize their architecture to succeed on the initial increments. 106753919 - 31 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 Figure 2-3. Critical off-Nominal Condition that Cannot Be Handled by the Nominal-Case Architecture This analysis enabled TRW not only to extend its risk management guidelines to cover critical off-nominal scenarios, but also to help convince future customers to reinterpret traditional waterfall milestones to include validation of feasibility evidence. A resulting example success story, the CCPDS-R project, is summarized in [Royce, 1998]. 2.9 Common Examples of Inadequate Evidence Common examples of inadequate evidence, or what could be referred to as non-evidence include these statements: 1. “Our engineers are tremendously creative. They will find a solution for this.” 2. “We have three algorithms that met the Key Performance Paramaters (KPPs) on small-scale nominal cases. At least one will scale up and handle the off-nominal cases.” 3. “We’ll build it and then tune it to satisfy the KPPs.” 4. “The COTS vendor assures us that they will have a security-certified version by the time we need to deliver.” 5. “We have demonstrated solutions for each piece from our NASA, Navy, and Air Force programs. It’s a simple matter of integration to put them together.” Numerous projects have repeated the consequences of proceeding without evidence that their plans and specifications are compatible and feasible. Why does this happen? Most often 106753919 - 32 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 because the project has been pushed into an inadequate budget and schedule, particularly for systems engineering, and because its progress payments and award fees are initially based on the existence of specifications rather than on their feasibility. As a result, one often hears statements such as the above in place of evidence at milestone reviews. What can be done about this? First, enough budget and schedule needs to be provided (models such as COSYSMO [Valerdi, 2005] are a good cross-check). Next, plans need to be made and progress monitored for both producing and reviewing the feasibility evidence. And if progress reviews uncover statements such as the above, corrective action such as the examples in the next section needs to be performed. 2.10 Examples of Making the Evidence Adequate These candidate actions in italics can resolve the issues of the previous section: 1. “Our engineers are tremendously creative. They will find a solution for this.” Have the creative engineers prototype and evaluate a solution on some key nominal and off-nominal scenarios. 2. “We have three algorithms that met the Key Performance Paramaters (KPPs) on small-scale nominal cases. At least one will scale up and handle the off-nominal cases.” Prototype and evaluate the three examples on some key nominal and off-nominal scenarios. 3. “We’ll build it and then tune it to satisfy the KPPs.” Develop prototypes and/or simulations and exercise them to show that the architecture will not break while scaling up or handling off-nominal cases. 4. “The COTS vendor assures us that they will have a security-certified version by the time we need to deliver.” Conduct a scaled-down security evaluation of the current COTS product. Determine this and other vendors’ track records for getting certified in the available time. Investigate alternative solutions. 5. “We have demonstrated solutions for each piece from our NASA, Navy, and Air Force programs. It’s a simple matter of integration to put them together.” Have a tiger team prototype and evaluate the results of the simple matter of integration. 2.11 ICM Detailed Activity Descriptions Stage II of the Incremental Commitment Life Cycle provides a framework for concurrent engineering and development of multiple increments. Note: The term “concurrent engineering” fell into disfavor when behind-schedule developers applied it to the practice of proceeding into development while the designers worked on finishing 106753919 - 33 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 the design. Not surprisingly, the developers encountered a high rework penalty for going into development with weak architecture and risk resolution. “Concurrent engineering” as applied in the ICM is much different. It is focused on doing a costeffective job of architecture and risk resolution in Stage I; and on performing stabilized development, verification, and validation of the current system increment while concurrently handling the systems change traffic and preparing a feasibility-validated architecture and set of plans for the next increment in Stage II. Figure 2-4 provides more detailed descriptions of the activities concurrently going on during the various ICM phases. A Development Commitment Review / Milestone B (DCR/MS-B) package contains all of the Foundations material (in risk-driven levels of detail) needed for its independent expert review preceding its DCR/MS-B: operations concept, requirements, architecture, plans, business case, feasibility evidence, and risk management plans as necessary. Even further detail on the contents of this material is provided next. 106753919 - 34 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 Figure 2-4. ICM Detailed Activity Descriptions 106753919 - 35 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs 2.12 Version 0.5 Focus of Each Commitment Review Commitment Reviews are so called because it reviews the evidence, created in the previous phase, and decides how to commit: proceed ahead, do more work, or discontinue the program. Evidence generally comes in the form of review packages, such as the Development Commitment Package (DC Package), created during the Foundations phase, used during the OCD, determining whether to proceed to the Development phase. These packages contain work products and feasibility evidence, such as prototypes, studies, estimates, bases of estimates. After reviewing the package and any other evidence, risks are determined. If the risks are acceptable or negligible, commit to the next phase. If the risks are high, but probably addressable, do more work within the current phase. If the risks are too high or unaddressable, then efforts should be discontinued altogether. 2.13 Lean Risk Management Plan The word “Plan” often conjures up a heavyweight document full of boilerplate and hard-toremember sections. Risk management plans should be particularly lean and risk-driven as illustrated in the outline presented in Figure 2-5. Risk Management Plan Outline 1. Objectives (the “Why”) 2. Deliverables and Milestones (the “What” and “When”) 3. Responsibilities (the “Who” and “Where”) 4. Approach (the “How”) 5. Resources (the “How Much”) Figure 2-5. Lean Risk Management Plan Outline The following is an example of how the outline might be used to develop a risk management plan to conduct fault tolerance prototyping to mitigate identified risks that the fault tolerance features might seriously compromise performance aspects such as throughput, real-time deadline satisfaction, and power consumption. It also shows the examples of the types of information to be provided for responsibilities, the risk management approach, and the needed funding resources. Objectives: Determine, reduce level of risk of the fault tolerance features causing unacceptable performance (e.g., throughput, response time, power consumption). Create a description of and a development plan for a set of low-risk fault tolerance features Deliverables and Milestones: By week 3: Evaluation of fault tolerance option, assessment of reusable components, draft workload characterization, evaluation plan for prototype exercise, and description of prototype. 106753919 - 36 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 By week 7: Operational prototype with key fault tolerance features; workload simulation; instrumentation and data reduction capabilities; and draft description, plan for fault tolerance features. By week 10: Evaluation and iteration of prototype; revised description, plan for fault tolerance features. Responsibilities: System Engineer: G. Smith—Tasks 1, 3, 4, 9, 11, support of tasks 5, 10 Lead Programmer: C. Lee—Tasks 5, 6, 7, 10 support of tasks 1, 3 Programmer: J. Wilson—Tasks 2, 8, support of tasks 5, 6, 7, 10 Approach: Design-to-Schedule prototyping effort; driven by hypotheses about fault toleranceperformance effects; using multicore processor, real-time OS, add prototype fault tolerance features. Evaluate performance with respect to representative workload. Refine Prototype based on results observed. Resources: $60K Full-time system engineer, lead programmer, programmer (10 weeks)*(3 staff)*($2K/staff-week) $0K 3 Dedicated workstations (from project pool) $0K 2 Target processors (from project pool) $0K 1 Test co-processor (from project pool) $10K Contingencies $70K Total Note that much of this risk plan information can be provided in a resource-loaded network tool. 2.14 FED Development Process Framework The appropriate level of detail for the contents of the FED are based on the perceived risks and criticality of the system to be developed. It is NOT a “one size fits all” process, but rather a framework to help developers and stakeholders determine the appropriate level of analysis and evaluation. It is also not limited to new system developments, but can be effective in major upgrades to existing systems or systems of systems. As with other ICM artifacts, FED process and content are risk-driven. Generic sets of steps are provided, but they need to be tailored to each situation. Tailoring occurs at increasing levels of detail in Exploration, Validation, and Foundations phases. Tailoring can be satisfied by pointers to existing evidence. Tailoring also applies to the Stage II Foundations rebaselining process Within the FED, representative scenarios, including critical off-noinal scenarios, should be provided for exercising large simulations, and testbed evaluation facilities, processes and evaluation criteria should also be prepared. 106753919 - 37 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs 2.15 Version 0.5 Steps for Developing Feasibility Evidence Table 2-1 outlines a process that can be used for developing feasibility evidence. The process clearly depends on having the appropriate work products for the phase (operational concept, requirements specification, architecture, etc.). As part of the engineering work, critical feasibility assurance issues are identified that are critical to the success of the system development program. These are the issues for which options are explored, as well as potentially viable options, further investigated with respect to cost, schedule, Return on Investment (ROI), etc. 106753919 - 38 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 Table 2-1. Steps for Developing FED Step Description Examples/Detail A. Develop phase work-products/artifacts See ICM Anchor Point Milestone Content charts provided in the chapter for each phase/anchor point milestone. B. Determine most critical feasibility assurance issues Issues for which lack of feasibility evidence is program-critical C. Evaluate feasibility assessment options Cost-effectiveness, risk reduction leverage/ROI, rework avoidance Tool, data, scenario availability D. Select options, develop feasibility assessment plans E. Prepare FED assessment plans and earned value milestones Try to relate earned value to risk-exposure avoided rather than budgeted cost F. Begin monitoring progress with respect to plans Also monitor project/technology/objectives changes and adapt plans G. Prepare evidence-generation enablers Assessment criteria Parametric models, parameter values, bases of estimate COTS assessment criteria and plans Benchmarking candidates, test cases Prototypes/simulations, evaluation plans, subjects, and scenarios Instrumentation, data analysis capabilities H. Perform pilot assessments; evaluate and iterate plans and enablers I. Assess readiness for Commitment Review J. Hold Commitment Review when ready; adjust plans based on review outcomes Shortfalls identified as risks and covered by risk mitigation plans Proceed to Commitment Review if ready NOTE: “Steps” denoted by letters rather than numbers to indicate that many are done concurrently. Also note that the FED activities need to be planned and assigned an appropriate earned value. The earned value should be based on the potential risk exposure costs, not the perceived 106753919 - 39 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 available budget. See Competitive Prototyping (Chapter 5) for additional guidance on how much “information” to buy via the feasibility assessment process. 2.16 Large-Scale Simulation and Testbed FED Preparation Example As seen in Figure 2-6, for large, complex systems, it is often very cost effective to begin development of a modeling and simulation testbed to support both the system infrastructure and application developers as well as those trying to evaluate the feasibility of the various alternatives. Figure 2-6. Large-Scale Simulation and Testbed FED Preparation Example 106753919 - 40 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs 2.17 Version 0.5 Example of FED Risk Evaluation Criteria In order to evaluate the findings of a feasibility assessment and determine the appropriate next steps, criteria are needed to help determine the associated risk for each finding. The following information can be used to assign a risk level to the various findings and guide the outcome of the associated Commitment Review: Negligible: Anticipated 0-5% budget and/or schedule overrun. Identified only minor shortfalls and imperfections expected to affect the delivered system. Low: Anticipated 5-10% budget and/or schedule overrun. Identified 1-3 moderate shortfalls and imperfections expected to affect the delivered system. Moderate: Anticipated 10-25% budget and/or schedule overrun. Identified >3 moderate shortfalls and imperfections expected to affect the delivered system. Major: Anticipated 25-50% budget and/or schedule overrun. Identified 1-3 mission-critical shortfalls and imperfections expected to affect the delivered system. Severe: Anticipated >50% budget and/or schedule overrun. Identified >3 mission-critical shortfalls and imperfections expected to affect the delivered system. 2.18 Conclusions Anchor Point milestones enable synchronization and stabilization of concurrent engineering. They have been successfully applied on small to large projects. APs also provide incremental stakeholder resource commitment points. The FED enables evidence of program feasibility to be evaluated at each of the APs and are produced by developer and then evaluated by the stakeholders and independent experts. To successfully develop a FED for each AP, the FED needs to be defined as a “first class” deliverable, it needs to be planned (with resources), and part of the program earned value management system. Any shortfalls in evidence are sources of uncertainty and risk, and should be covered by risk management plans. And finally, the FED and associated risk management plan(s) should be part of the traditional milestone content and reviews. 2.19 References B. Boehm and W. Hansen, “The Spiral Model as a Tool for Evolutionary Acquisition,” CrossTalk, May 2001. B. Boehm, A.W. Brown, V. Basili, and R. Turner, “Spiral Acquisition of Software-Intensive Systems of Systems,” CrossTalk, May 2004, pp. 4-9. B. Boehm and J. Lane, “Using the ICM to Integrate System Acquisition, Systems Engineering, and Software Engineering,” CrossTalk, October 2007, pp. 4-9. J. Maranzano et. al., “Architecture Reviews: Practice and Experience,” IEEE Software, March/April 2005. R. Pew and A. Mavor, Human-System Integration in the System Development Process: A New Look, National Academy Press, 2007. 106753919 - 41 - Version Date: 12/31/08 Chapter 2: DoD-ICM Fundamentals – AP Milestones & FEDs Version 0.5 W. Royce, Software Project Management, Addison Wesley, 1998. RQ-4A/B Global Hawk High Altitude, Long Endurance, Unmanned Reconnaissance Aircraft, USA, http://www.airforce-technology.com/projects/global/, accessed on 8 July 2008. R. Valerdi, “The Constructive Systems Engineering Cost Model,” Ph.D. dissertation, USC, August 2005. CrossTalk articles: http://www.stsc.hill.af.mil/crosstalk 106753919 - 42 - Version Date: 12/31/08 Chapter 3: ICM Fundamentals – Incremental Adoptability Version 0.5 3Chapter 3: ICM Fundamentals – Incremental Adoptability 3.1 Introduction Most system development programs are not new developments, but are currently focused on maintaining existing capabilities, enhancing existing capabilities, or incorporating new capabilities. For these existing programs, it is often not possible to make major process changes all at once. Rather, it is better to incrementally change processes over time. Existing programs may benefit from some ICM principles and practices, but not others. Problem programs may find some ICM practices helpful in recovering viability. There are primary opportunities for incremental adoption of ICM principles and practices, this chapter deals with many of them. 3.2 Supplementing Traditional Requirements and Design Reviews with Development and Review of Feasibility Evidence [Further elaboration TBD] 3.3 Stabilized Incremental Development and Concurrent Architecture Rebaselining [Further elaboration TBD] 3.4 Using Schedule as Independent Variable and Prioritizing Features to be Delivered [Further elaboration TBD] 3.5 Continuous Verification and Validation [Further elaboration TBD] 3.6 Using the Process Decision Table [Further elaboration TBD] 106753919 - 43 - Version Date: 12/31/08 Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase Version 0.5 4Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase 4.1 Introduction The ICM is the only current life cycle process model that follows DoDI 5000.02 in beginning with User Needs and Technology Opportunities. Programs using process models that start with pinning down Requirements often run into trouble by overfocusing on requirements for materiel users and overlooking other success-critical stakeholders (such as maintainers and interoperators) and their needs; by prematurely imposing requirements constraints that disqualify superior solution opportunities (such as commercial-off-the-shelf (COTS) or contractor capabilities, as in the Section 2-3 example); or by quickly converging on requirements that lead to obsolete solutions, by neglecting growth trends in user needs or by basing requirements on technologies with limited growth potential (such as inability to accommodate workload growth or system-ofsystems interoperability). Figure 4-1 shows the nature and location of the Needs and Opportunities phase in the DoD life cycle. The main objective of this phase is to develop evidence that enables the Milestone Decision Authority to correctly decide whether a materiel solution is appropriate to pursue. This is done in the ICM based on the level of risk involved in making the decision to proceed, as shown in the “Risk?” circle at the bottom of the Needs and Opportunities phase column in Figure 4-1. If the evidence indicates an acceptable level of risk in going forward toward a materiel solution, the upward-right arrow passes the project into the next Materiel Solution Analysis and Analysis of Alternatives phase. If the evidence points to a clear decision that a non-materiel solution is best, the downward arrow is followed and the materiel solution process is discontinued, or perhaps rescoped to explore an alternative mission scope for which a materiel solution may be useful and appropriate. If the evidence to support a clear decision is insufficient, the upward-left arrow is taken and further exploration of needs and opportunities is pursued. For very small systems for which an alreadyowned COTS product can be tailored to produce the solution, the risk of going directly into Development is negligible, and the right-pointing arrows can be taken to skip phases, but this is not the case for major defense acquisition programs (MDAPs). For MDAPs, a non-trivial level of investment by several involved stakeholders is needed to produce the required evidence to support a Materiel Development Decision. These stakeholders generally want some level of evidence that this investment will be worthwhile for them. Thus, the Needs and Opportunities phase begins with a similar evidence/risk-based decision to proceed into the Needs and Opportunities phase, based on evidence provided by the Service or Agency desiring to pursue a materiel-development acquisition process. 106753919 - 44 - Version Date: 12/31/08 Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase Version 0.5 Figure 4-1. Overview of the ICM - Exploration Highlighted The primary Activities to be concurrently performed in a risk- and opportunity-driven process in the Needs and Opportunities phase (and later phases) are shown in Figure 4-2. This figure shows that Understanding Needs and Envisioning Opportunities are key activities to be performed at the beginning of the DoD life cycle process. Other key activities shown in Figure 4-2 are exploring the level of need and scoping of a materiel solution; the identification and prioritization of its goals and objectives; the characterization of its legacy system or interoperability constraints; the architectural compatibility of its candidate hardware, software, and human factors elements; and its affordability based on early life-cycle planning and estimating efforts. Evidence of the feasibility of the candidate materiel solution is provided by some combination of prototyping, modeling, simulation, benchmarking, and cost-schedule analysis, and used to enable stakeholders to assess their risks of proceeding and to negotiate mutual commitments to invest further resources in proceeding or continuing to develop adequate evidence of feasibility. 106753919 - 45 - Version Date: 12/31/08 Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase Version 0.5 Figure 4-2. ICM Activity Categories and Level of Effort 4.2 Phase Key Elements Table 4-1 identifies the key phase elements in the Needs and Opportunities phase. Each element will be elaborated in a section below. 106753919 - 46 - Version Date: 12/31/08 Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase Version 0.5 Table 4-1. Needs and Opportunities (Exploration) Goals & Objectives Prepare for MDD Develop ICD: OpCon, Needed Capabilities, MSR Identify SCSHs Identify Common-Case or alternative life cycle process Evidence of Feasibility, Risk Addressal Next-phase plans SCSH commitment to way forward Entry Conditions Committed protagonist Phase plan Adequate resources to achieve goals and objectives: funds, time, staff, and info sources Fit to enterprise strategy Inputs Steps (concurrent) Exit Conditions Work Products Pitfalls Top-level Understand needs: Validated shared MSR: Evidence Inadequate phase phase plan and rationale Funds, time, staff, info sources SCSH support for phase current and future gaps, DOTMLPF aspects Explore opportunities Identify enabling initiatives and SCSHs Determine SCSH objectives and priorities Identify feasible vision package: Benefits chain & SCSHs; System boundary, OpCon; Key solution elements; and FED (including business case) Validated MSR, approved ICD solution region Evaluate MDSs, nonMDSs Identify Common-Case or alternative life cycle process Develop and validate shared vision elements Prepare MSR, ICD Obtain SCSH decision commitments for next phase Common-Case or alternative life cycle process Feasibility Evidence or risk-management plans CP strategy or candidate alternatives to analyze Detailed next-phase plans Committed to by that needs are critical; non-MDS solutions are inadequate; at least one MDS is feasible Top-level shared vision package Associated risk mitigation plans Common-Case or alternative life cycle process SCSH Commitment MOAs Identification of next-level SCSHs, IPT structure Detailed nextphase plans, ICD plans, budget and schedule Neglecting SCSHs Shortfalls in FEDs or SCSH MOAs without riskmanagement plans Poor project fit to enterprise strategy Overconstrained and incompatible quality factors Shortfalls in CP strategy Cursory tech readiness analysis, analysis of nonMDS inadequacy Inadequate nextphase plans, budget, schedule SCSHs CP = Competitive Prototyping DOTMLPF = Doctrine, Organization Training, Materiel, Leadership and education, Personnel, Facilities FED = Feasibility Evidence Description ICD = Initial Capabilities Document MDD = Materiel Development Decision MDS = Materiel Development Solution MOA = Memorandum of Agreement MSR = Materiel Solution Rationale OpCon = Operational Concept SCSH = Success-Critical Stakeholder 106753919 - 47 - Version Date: 12/31/08 Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase 4.3 Version 0.5 Goals and Objectives 4.3.1 Prepare for Material Development Decision 4.3.2 Develop Initial Capabilities Document 4.3.3 Identify the Success-Critical Stakeholders 4.3.4 Identify Common-Case or Alternative Life Cycle Process 4.3.5 Evidence of Feasibility, Risk Addressal 4.3.6 Next-Phase Plans 4.3.7 SCSH Commitment to Way Forward 4.4 Entry Conditions 4.4.1 Committed Protagonist 4.4.2 Phase Plan 4.4.3 Adequate Resources to Achieve Goals and Objectives 4.4.4 Fit to Enterprise Strategy 4.5 Inputs 4.5.1 Top-Level Phase Plan and Rationale 4.5.2 Funds, Time, Staff, Info Sources 4.5.3 Success-Critical Stakeholder Support for Phase 4.6 Steps (Concurrent) 4.6.1 Understand Needs 4.6.2 Explore Opportunities 4.6.3 Identify Enabling Initiatives and Success-Critical Stakeholders 106753919 - 48 - Version Date: 12/31/08 Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase Version 0.5 4.6.4 Determine Success-Critical Stakeholder Objectives and Priorities 4.6.5 Identify Feasible Solution Region 4.6.6 Evaluate Materiel Development Solutions, Non-Materiel Development Solutions 4.6.7 Identify Common-Case or Alternative Life Cycle Process 4.6.8 Develop and Validate Shared Vision Elements 4.6.9 Prepare Materiel Solution Rationale, Initial Capabilities Document 4.6.10 Obtain Success-Critical Stakeholder Decision Commitments for Next Phase 4.7 Exit Conditions 4.7.1 Validated Shared Vision Package 4.7.2 Validated Materiel Solution Rationale, Approved Initial Capabilities Document 4.7.3 Common-Case or Alternative Life Cycle Process 4.7.4 Feasibility Evidence or Risk-Management Plans 4.7.5 CP Strategy or Candidate Alternatives to Analyze 4.7.6 Detailed Next-Phase Plans 4.7.7 Committed to by Success-Critical Stakeholders 4.8 Work Products 4.8.1 Materiel Solution Rationale 4.8.2 Top-Level Shared Vision Package 4.8.3 Associated Risk Mitigation Plans 4.8.4 Common-Case or Alternative Life Cycle Process 106753919 - 49 - Version Date: 12/31/08 Chapter 4: Stage I: Incremental Definition – The Needs and Opportunities/Exploration Phase Version 0.5 4.8.5 Success-Critical Stakeholder Commitment Memoranda of Agreement 4.8.6 Identification of Next-Level Success-Critical Stakeholders, Integrated Product Team Structure 4.8.7 Detailed Next-Phase Plans, Initial Capabilities Document 4.9 Pitfalls 4.9.1 Inadequate Phase Plans, Budget and Schedule 4.9.2 Neglecting Success-Critical Stakeholders 4.9.3 Shortfalls in Feasibility Evidence Descriptions or Success-Critical Stakeholder Memoranda of Agreements Without RiskManagement Plans 4.9.4 Poor Project Fit to Enterprise Strategy 4.9.5 Overconstrained and Incompatible Quality Factors 4.9.6 Shortfalls in Competitive Prototyping Strategy 4.9.7 Cursory Technical Readiness Analysis, Analysis of Non-Materiel Development Solution Inadequacy 4.9.8 Inadequate Next-Phase Plans, Budget, Schedules 106753919 - 50 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table Version 0.5 5Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table 5.1 Introduction As with other models trying to address a wide variety of situations, the general form of the Incremental Commitment Model is rather complex. However, its risk-driven nature enables projects to determine a set of twelve common risk patterns and organize them into a decision table that can help new projects converge on a process that fits well with their particular process drivers. For each of these twelve special cases, the ICM Process Decision Table provides toplevel guidelines for tailoring the key activities of the ICM, along with suggested lengths between each internal system build and each external system increment delivery. The process drivers used as inputs to the Decision Table include: the system’s size and complexity, its rate of change, its mission-criticality, the extent of non-developmental item (NDI) support for its desired capabilities, and the capability of available organizational and personnel for developing the system. The Decision Table includes twelve common risk-driven special cases of the ICM. With representative examples in parentheses, these are (1) Use NDI (Small accounting system), (2) Agile (Small e-services), (3) Scrum of Scrums (Business data processing), (4) Formal methods (LSI chip, security kernel), (5) Software-embedded hardware component (Multi-sensor control device), (6) Indivisible initial operational capability (Complete vehicle platform), (7) NDIintensive system (Supply chain management), (8) Hybrid agile/plan-driven system (C4ISR system), (9) Multi-owner system of systems (SoS) (Net-centric emergency services), (10) Family of systems (Medical device product line), (11) Brownfield (obsolete C4ISR system replacement), and (12) Net-Centric Services (a: community support; b: quick response decision support). In addition, some of the larger special cases can be architected to enable some of their components to be treated more simply as one of the smaller special cases. Often, the top-level choice of special case can be made by the end of the ICM Needs and Opportunities/Exploration Phase, with the lower-level special case decisions made in subsequent phases. 5.2 The ICM Process Decision Table In many situations, the explorations in the Needs and Opportunities phase discussed in Chapter 4 will show that a special-case process opportunity exists that will satisfy the users’ needs without the full set of phases and activities in ICM. These cases cover the very small to the very large as well as the use of commercial off-the-shelf (COTS) software products to the development of a large, complex custom software application or integrated sets of software applications. Each of these situations presents risks at the various stages of development, some risks more critical than others. The goal of the ICM is to identify these risks and then tailor the process to include rigor where necessary to investigate and manage the risks and to streamline the process when risks are negligible, allowing the development team to be more agile when possible. Table 5-1 contains a list of the special cases of the ICM and an example of each case. These cases cover a broad spectrum of systems. In addition, Table 1 describes the characteristics of the case (or application 106753919 - 51 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table Version 0.5 category) with respect to size and complexity; expected change rate during the development process; the overall criticality of the application, typically as it applies to human life, security/protection of information, or financial liability; and the type of NDI support one might expect for an application in this category. It also indicates the organizational and personally capabilities that are required to be successful at building this type of application and the typical/suggested lengths for each internal system build and each external system increment delivery. Table 5-2 provides top-level guidelines for tailoring the key activities in Stage I (incremental definition) and Stage II (incremental development and operations) of the ICM. 106753919 - 52 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table Version 0.5 2. Agile 1-30 Low-Med Business data processing Security kernel; Safety-critical LSI chip Multi-sensor control device Med 1-10 Med-High Low 0.3 Extra High Low 0.3-1 Complete vehicle platform Supply chain management MedHigh 8. Hybrid agile/ plan-driven system 9. Multi-owner system of systems 5. HW with embedded SW component 6. Indivisible IOC 7. NDI- intensive 10. Family of systems 11. Brownfield 12a. Net- Centric Services— Community Support Time/Build; Time/Increment Complete Low 3. Architected Agile 4. Formal Methods Organizational and Personnel Capability Small accounting E-services NDI Support 1. Use NDI Criticality Example Change Rate (%/Month) Special Case Size, Complexity Table 5-1. Characteristics of the Risk-Driven Special Cases of the ICM Good; in place Good; most in place None Agile-ready Med-high Agile-ready Med-high Strong formal methods experience <= 1 day; 2-6 weeks 2-4 weeks; 2-6 months 1-5 days; 1-4 weeks Med-Very High Good; in place Experienced; med-high SW: 1-5 days; Market-driven 0.3-1 HighVery High Some in place Experienced; med-high MedHigh 0.3-3 Med-Very High NDI-driven architecture C4ISR system MedVery High Mixed parts; 1-10 Mixed parts Net-centric military operations Medical device product line Incremental legacy phaseout Community Services or Special Interest Group Response to competitor initiative Very High Mixed parts; 1-10 1-3 Mixed parts; Med-Very High Very High NDIexperienced; med-high Mixed parts SW: 2-6 weeks; Platform: 6-18 months SW: 1-4 weeks; Systems: 6-18 months 1-2 months; 9-18 months MedVery High HighVery High LowMed Med-Very High 0.3-3 Med-High 0.3-3 Low-Med Many NDIs; some in place Some in place NDI as legacy replacement Tailorable service elements Related experience, med-high Related experience, med-high Legacy reengineering NDIexperienced 2-4 months; 18-24 months 1-2 months; 9-18 months 2-6 weeks/ refactor ; 2-6 months <= 1 day; 6-12 months 12b. Net-Centric Med3-30 Med-High Tailorable NDI<= 1 day ; Services—Quick High service experienced QR-driven Response elements Decision Suppport C4ISR: Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance; HW: Hardware; IOC: Initial Operational Capability; NDI: Non-Development Item; QR : Quick Response ; SW: Software. 106753919 - 53 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table Version 0.5 Table 5-2. Key Activities for Each Special Case of the ICM Special Case Example 1. Use NDI Small accounting E-services 2. Agile 3. Architected Agile Business data processing 4. Formal Methods Security kernel; Safety-critical LSI chip Multi-sensor control device 5. HW with embedded SW component 6. Indivisible IOC Complete vehicle platform 7. NDI- intensive Supply chain management 8. Hybrid agile/ plan-driven system C4ISR system 9. Multi-owner system of systems Net-centric military operations Medical device product line 10. Family of systems 11. Brownfield 12a. Net-Centric Services— Community Support Incremental legacy phaseout Community Services or Special Interest Group Response to competitor initiative Key Stage I Activities: Incremental Definition Acquire NDI Key Stage II Activities: Incremental Development, Operations Use NDI Skip Valuation, Architecting phases Combination Valuation, Architecting phases. Complete NDI preparation Precise formal specification Scrum plus agile methods of choice Architecture-based Scrum of Scrums Concurrent HW/SW engineering. CDR-level ICM DCR Determine minimum-IOC likely, conservative cost. Add deferrable SW features as risk reserve Thorough NDI-suite life cycle cost-benefit analysis, selection, concurrent requirements/ architecture definition Full ICM; encapsulated agile in high change, low-medium criticality parts (Often HMI, external interfaces) Full ICM; extensive multiowner team building, negotiation Full ICM; Full stakeholder participation in product line scoping. Strong business case Re-engineer/refactor legacy into services Filter, select, compose, tailor NDI IOC Development, LRIP, FRP. Concurrent version N+1 engineering Drop deferrable features to meet conservative cost. Strong award fee for features not dropped Formally-based programming language; formal verification Pro-active NDI evolution influencing, NDI upgrade synchronization Full ICM, three-team incremental development, concurrent V&V, nextincrement rebaselining Full ICM; large ongoing system/software engineering effort Full ICM. Extra resources for first system, version control, multi-stakeholder support Incremental legacy phaseout Evolve tailoring to meet community needs 12b. Net-Centric Filer, select, compose, tailor Satisfy quick response; evolve or Services—Quick NDI phase out Response Decision Suppport C4ISR: Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance; CDR: Critical Design Review DCR: Development Commitment Review FRP: Full-Rate Production HMI: Human-Machine Interface HW: Hardware IOC: Initial Operational Capability LRIP: Low-Rate Initial Production NDI: Non-Development Item SW: Software V&V: Verification and Validation. 106753919 - 54 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table 5.3 Version 0.5 Using the Process Decision Table Selecting and Tailoring a Decision Table Case. The general ICM chart has many decision options available, but its risk-driven approach results in most projects having simpler special cases that can usually be determined during the project’s Exploration phase. The decision table indicates the most common special cases. But it is not exhaustive. If your risk pattern is not included, look at the closest approximation to it, and think through an appropriate risk-driven variant of it. Even if your pattern is included in the table, there may be special aspects of your project that require additional tailoring (e.g., multiple cultures, distant time zones, legacy constraints). The decision table should be a stimulus to thinking, not a substitute for it. In addition, many complex projects have simpler pieces in them. Rather than apply a one-size-fits-all process to all of them, often you can architect the system to use simpler processes on the simpler parts. The rest of this section describes each of the ICM special cases and typical risks associated with each case. 5.3.1 Case 1: Use NDI (Non-Development Items) Suppose you have an application for which an appropriate NDI (COTS, open source, reuse library, customer-furnished package) solution is available, and other options of developing perhaps a better version yourself or outsourcing such a development. Even if you produce a better solution (frequently not the case), you will generally incur more expense and take longer to begin to capitalize on its benefits. And you will often find that the NDI package has features that you hadn’t realized you would need, but are there when you need them. On the other hand, there are risks that may disqualify some NDIs. They may be overly complex for your needs, incompatible with your other applications, or highly volatile. See the discussion in Section 33.1 of Software Engineering Economics (Boehm, 1981) for more about the pros and cons of NDI solutions. 5.3.2 Case 2: Pure Agile Methods If your project is small (less than 10 people) and its criticality involves the loss of discretionary vs. essential funds, a pure agile method such as Extreme Programming (Beck, 1999), Scrum (Schwaber, 2002), or Crystal (Cockburn, 2002) is generally best if you have relatively highcapability, agile-ready personnel. The risks of using a more formal, plan-driven approach are less adaptability to change and belated feedback on the product’s capabilities. The biggest risk is to try to develop the application all at once for several months, and then find that it is a mismatch to the users’ needs. Agile developers have found that the best way to address this risk is to organize the project into short (2-6 week) delivery increments that may be incomplete, but provide early useful capabilities. Any flaws can then be detected early and fixed in the next increment (for very high criticality applications, this would not be acceptable). On a small project it is also easy to set up a daily build and regression test structure that identifies integration problems early when they are easier to fix. Some risks of using the agile approach is that it is harder to write a contract for what is being developed; that it may sub optimize for early success by using unscalable capabilities (fourth106753919 - 55 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table Version 0.5 generation languages, running all in main memory); or high personnel turnover. See (Boehm and Turner, 2004), pp. 121-128 for an example risk analysis of an agent-based event planning system, ending with a decision to go agile. 5.3.3 Case 3: Architected Agile For medium-size (20-80 people), medium complexity (reasonably mature and scalable technology; largely compatible shareholders), agile methods can be scaled using an Architected Agile approach with early investment in a largely change-prescient architecture and user/developer/customer team building. For relatively stable projects (0.3-1% change/month), plan-driven methods can be used with low risk. But for higher rates of changes (1-10%/month), a more agile approach is less risky. A risk analysis of a 50-person, medium sized architecturebased agile supply chain management project is provided on pages 106-121 of (Boehm and Turner, 2004). A number of organizations in such areas as corporate infrastructure, medical, aerospace, and ERP applications have reported significant gains in adaptability and quality of the Architected Agile approach over plan-driven methods for such projects. However, others that had less capable and agile-ready people, less management and customer commitment, and less up-front architecture investment have not. (Boehm, 2007) 5.3.4 Case 4: Formal Methods Formal methods involve the development and verification of a precise mathematical specification of the behavior of a hardware and/of software systems; an implementation of the system in formal-semantics-based languages; and a mathematical proof that the implementation is exactly equivalent to the specification (no more; no less). Such methods are expensive relative to standard commercial practice and require scarce high-capability personnel, but are inexpensive relative to a massive product recall, expensive lawsuits, or a massive loss of valuable and confidential information. Current formal methods generally require repeating the expensive mathematical proof process whenever the specification or implementation is changed, making the approach less viable for systems with highly volatile requirements. In general, non-developmental items are not precisely specified and verified enough to be safely used in such systems. Also, formal methods have limited scalability; almost all fully-verified software systems have less than 10,000 lines of code. However, some progress is being made toward modularizing such systems so that implementations and proofs can be built up incrementally via lower-level lemmas and theorems. 5.3.5 Case 5: Software Embedded Hardware Component The application classes above have been mostly software-intensive. The differences in economic and risk patterns for hardware-intensive projects (Boehm and Lane, 2007) will create different risk-based special cases of the ICM. Once a project commits to a particular manufacturing approach and infrastructure, the hardware cost of change will be much higher. And the primary costs at risk are in development and manufacturing, particularly if the component is cheaper to replace than to fix or if fixes can be accomplished by software workarounds. Thus, the ICM Stage I activities of producing and validating detailed specifications and plans are much more cost-effective than for the agile cases above. They will often go beyond the usual level of detail (Critical Design Review versus evidence-based Preliminary Design Review) for an ICM Development Commitment Review (DCR), since so many of the details are likely to be major 106753919 - 56 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table Version 0.5 sources of rework expenditures and delay if wrong. And after Initial Operational Capability (IOC) development, the rework risks generally dictate an incremental low rate initial production phase before a full-rate production phase in Stage II. In case the component is planned to evolve to subsequent hardware-software reconfigurations, the ICM approach of having a concurrent systems engineering team developing specifications and plans for the “N+1st “ increment could be adopted. The length of these later increments will be driven by the product’s marketplace or competitive situation. 5.3.6 Case 6: Indivisible IOC More complex hardware intensive systems, such as aircraft, may have a great deal of software that can be incrementally developed and tested, but may have an indivisible hardware IOC that must be developed as a unit before it can be safely tested (e.g., an automobile or aircraft’s braking system or full set of safety-critical vehicle controls). Relative to the cost and duration of the software increments, the indivisible hardware IOC has two significant sources of risk: It cannot fit into smaller software increments It cannot drop required hardware features to meet IOC schedule/cost/quality as independent variable. The first can be addressed by synchronizing the hardware IOC with the Nth software increment (e.g., (Rechtin and Maier, 1997) osculating orbits for hardware and software). If some combination of schedule/cost/quality is truly the project IOC’s independent variable, then one does not want to commit to a most-likely combination of schedule/cost/quality, as there will be a roughly 50% chance of an overrun or quality shortfall in completing the indivisible IOC. Rather, it is better to determine a conservative IOC cost and schedule for meeting the quality objective, and use the difference between the conservative cost and schedule and the most-likely cost and schedule as a risk reserve. The best way to do this is to use the conservative cost and schedule as the IOC target, and to determine a set of desired but non-essential, easy to drop software features that could be developed within the risk reserve. Then, if the most-likely indivisible IOC capability begins to overrun, some desired software features can be dropped without missing the scheduled delivery time and cost. There should also be a strong award-fee incentive for the developers to minimize the number of features that need to be dropped. 5.3.7 Case 7: NDI-Intensive Our experiences in developing USC web-service applications between 1996 and 2004 was that they went from 28% of the application’s functionality being delivered by NDI components to 80% (Yang et al, 2005). A similar trend was identified by the 2001 Standish Report, which reported that 53% of the functionality of commercial software applications was being delivered by NDI components in 2000 (Standish, 2001). The economics of NDI-intensive systems dictates a bottom-up versus a top-down approach to system development, in which the capability envelope of the NDI determines the affordable requirements, rather than a top-down requirements-to-capability approach. A large supply-chain management system may need to choose among several NDI candidates each for such functions as inventory control, trend analysis, supplier/customer relations management, transportation planning, manufacturing control, and financial transactions; and evaluate not only the candidates’ cost/performance aspects, but also their interoperability with each other and with the corporation’s legacy 106753919 - 57 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table Version 0.5 infrastructure. Besides NDI assessment, other significant sources of effort can be NDI tailoring, NDI integration, and effect of NDI version volatility and obsolescence; see (Yang et al, 2005). A particular challenge in Stage II is the effect of NDI volatility and obsolescence. Surveys have indicated that commercial NDI products have a new release about every 10 months, and that old releases are supported by the vendor for about 3 releases. Some large systems have had about 120 NDI components, indicating that about 12 components will have new releases each month, and that not upgrading will leave each component unsupported in about 30 months. In such cases, a great deal of attention needs to be paid to upgrade synchronization, and to pro-active NDI evolution influencing. Some large organizations synchronize their NDI upgrades to their major re-training cycles of about 12-18 months. For additional NDI best practices, see (Wallnau et al, 2002). 5.3.8 Case 8: Hybrid Agile/Plan-Driven System Some large, user-intensive systems such as C4ISR systems, air traffic control systems, and network control systems have a mix of relatively stable, high-criticality elements (sensors and communications; key business logic) and more volatile, more moderately critical elements such as GUI displays, electronic warfare countermeasures, and interfaces with externally evolving systems. As many acquisitions of this nature have shown, it is highly risky to apply one-sizefits-all processes and incentive structures to these differing classes of elements. Instead, in Stage I, it is important to determine which system elements belong in which class along with the other functions of understanding needs, envisioning opportunities, NDI assessment, scoping the system, and determining feasible requirements and increments performed for complex systems in Stage I; to architect the system to encapsulate the volatile parts for development by agile teams; and to organize to use plan-driven development for the other elements of the overall system. In Stage II, the cycles for the agile teams are likely to be significantly shorter than those for the plan-driven teams. 5.3.9 Case 9: Multi-Owner System of System In this situation, your goal is to integrate a set of existing systems (or guide and evolve the integration of a set of existing systems). These systems are primarily developed, owned, and maintained by an organization other than the one that is attempting to manage and guide the set of systems as a system of systems. Because of the independence of these constituent systems, the SoS organization has little or no formal control over the processes used to maintain and evolve the constituent systems. The SoS may be an enterprise wide business SoS, with the constituents being primarily COTS products along with some legacy applications. Or it may be Department of Defense (DoD) warfighting SoS, where the constituent legacy systems are integrated to increase capabilities on the battle field. Traditional systems engineering (SE) activities are typically tailored for the SoS case to define an SoS architecture, better coordinate the activities of multiple systems in migrating to the SoS architecture, and provide synchronization points for the SoS. In (OUSD AT&L, 2008), pilot studies have shown that many DoD SoS have re-organized the traditional SE activities into a set of seven core elements: 1) translating capability objectives, 2) understanding systems and relationships, 3) assessing performance to capability objectives, 4) developing, evolving, and maintaining SoS design, 5) monitoring and assessing changes, 6) addressing new requirements 106753919 - 58 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table Version 0.5 and options, and 7) orchestrating updates to SoS. Further analysis shows, that these elements map fairly well to the hybrid agile/plan-driven case (Case 8) at the SoS level. What makes this case different from Case 8, is that each of the constituent systems is using their own processes, which could be any of the above cases, depending on the scope, complexity, and characteristics of the constituent system. What is key is that the Case 9 extends Case 8 to include information or participation of the constituent systems in the agile planning activities and lets the “battle rhythm” of the constituent system increments guide the SoS plan-driven and V&V activities. 5.3.10 Case 10: Family of Systems Families of systems are typically a set of systems that belong to a product line and can be easily used to customize a solution for a given need. This might be a suite of medical devices or a suite of applications to customer support. This is often the set of systems developed by a vendor that become the NDI components for CASE 7 above. Again, the rigor required for the SoS case is present here. However, in this situation, the family of systems is typically owned and evolved by a single organization/vendor and presents a case where the owning organization has much more control over the evolution of the components of the family of systems, thus possibly reducing some risks and allowing the ICM process to be a little more streamlined. 5.3.11 Case 11: Brownfield A Brownfield counterexample involved a major US corporation that used a Greenfield systems engineering and development approach to develop a new Central Corporate Financial System to replace a patched-together collection of strongly-coupled and poorly documented COBOL business data processing programs. The system included an early Enterprise Resource Planning (ERP) system that was well matched to the corporation's overall financial needs, but not to its detailed business processes, which included various workarounds to compensate for difficulties with the legacy software. The new system was well organized to support incremental implementation, but was dropped at a cost of $40 million after two failed tries to provide continuity of service, due primarily to the infeasibility of incrementally phasing out the legacy software and business processes compatibly with the new system's incremental capabilities. The ICM can be used to organize systems engineering and acquisition processes in ways that better support Brownfield legacy software re-engineering so that organizations can better provide continuity of services. In particular, its concurrent activities of Understanding Needs, Envisioning Opportunities, System Scoping and Architecting, Feasibility Evidence Development, and Risk/Opportunity Assessment enable projects to focus specifically on their legacy system constraints and on opportunities to deal with them. The application of the ICM to the Brownfield corporate counterexample situation would have avoided the failures of the Greenfield development. Its Understanding Needs activity would have determined the ways in which the legacy system had intertwined financial and other business services. For examples, Project Services included budgeting and scheduling, work breakdown system accounting, earned value management intertwined with requirements, version and configuration management; Contract Services included expenditure category management, billing, and receivables management intertwined with deliverables management and engineering 106753919 - 59 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table Version 0.5 change proposal tracking; with similar intertwining in Personnel Services and Marketing Services. The ICM's Envisioning Opportunities activity would have identified opportunities to use largescale refactoring methods to decouple the financial and non-financial elements of these services in ways that would make it feasible for them to interoperate with a Financial Services element. Its System Scoping and Architecting activity would have repackaged the legacy software and developed the new architecture around financial services that would support incremental phaseout, and its Feasibility Evidence Development activity would have assessed any outstanding risks and covered them with risk management plans that would be tracked to ensure project success. A good example of a Brownfield methodology with several case study examples is provided in (Hopkins and Jenkins, 2008). 5.3.12 Case 12a: Net-Centric Services—Community Support Net-Centric Services for community support tend to fall into two categories. One involves community service organizations providing needed services for child care, elder care, handicapped, homeless, or jobless people. Their information processing service needs include tracking of clients, volunteers, donors, donations, events, and provision of news and communication services. These used to require considerable programming to realize, but now can be realized by combining and tailoring NDI packages offering such services. The other category of net-centric services for community support involves special interest groups (professionals, hobbyists, committees, ethnic groups) with similar needs for tracking members, interests, subgroups, events, and provision of news and communication services. Development of such capabilities involves much more prototyping and much less documentation than is needed for more programming-oriented applications; for example, the internals of the NDI packages are generally unavailable for architectural documentation, and the resulting system capabilities are more driven by NDI capabilities than by prespecified requirements documents. A good example of how the ICM provides support for such net-centric services is provided in (Boehm and Bhuta, 2008) and the entire journal provides additional approaches and case studies of net-centric services development. 5.3.13 Case 12b: Net-Centric Services: Quick Response Decision Support Another form of net-centric services involves rapidly configuring a capability to analyze alternative decision options in response to a competitor's initiative. These might involve a competitor beginning to penetrate a business's home territory, and might involve the need to evaluate the relative costs and benefits of alternative strategies of expanding services, expanding service locations, or repricing products or services. NDI packages to help analyze geographical, logistical, human resources, and financial implications of alternative competitive response decisions can be rapidly configured to provide a quick-response capability for converging on a decision. Once the decision is made, the resulting capability may be phased out, or evolved into a more general and maintainable capability for similar future situations. Again, the November/December 2008 special issue of IEEE Software on "Opportunistic Software Systems 106753919 - 60 - Version Date: 12/31/08 Chapter 5: Stage I: Incremental Definition – The ICM Process Decision Table Version 0.5 Development" (Boehm and Bhuta, 2008) provides additional perspectives on this area of netcentric services development. 5.4 References Beck, K. 1999. Extreme programming explained, Addison Wesley. Boehm, B. 1981. Software engineering economics, Prentice Hall. Boehm, B. 2007. Agility and quality”, Keynote Presentation, ICSE 2007 Workshop on Software Quality. Boehm, B. and Bhuta, J. 2008. Balancing Risks and Opportunities in Component-Based Software Development. IEEE Software, November/December 2008, pp. 56-63. Boehm, B. and Lane, J. 2007. Using the incremental commitment model to integrate system acquisition, systems engineering, and software engineering, CrossTalk, Vol. 20, No. 10. Boehm, B. and Turner, R. 2004. Balancing agility and discipline: a guide for the perplexed. Addison-Wesley, Boston. Cockburn, A. 2002. Agile software development, Addison Wesley. Hopkins, R. and Jenkins, K. 2008. Eating the IT Elephant: Moving from Greenfield Development to Brownfield, IBM Press, 2008 Marenzano, J. et al. 2005. Architecture reviews: practice and experience. IEEE Software, March/April 2005, pp. 34-43. Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (OUSD AT&L), 2008. Systems engineering for systems of systems, Version 1.0. Washington, DC: Pentagon. Pew, R. W., and Mavor, A. S. 2007. Human-System Integration in the System Development Process: A New Look. National Academy Press. Schwaber, K. and Beedle, M. 2002. Agile Software Development with Scrum. Prentice Hall. Standish Group 2001. Extreme Chaos. Wallnau, K., Hissam, S., and Seacord, R. 2002. Building Systems from Commercial Components. Addison Wesley. 106753919 - 61 - Version Date: 12/31/08 Chapter 6: Stage I: Incremental Definition – Materiel Solution Analysis & AoA/Valuation Phase Version 0.5 6Chapter 6: Stage I: Incremental Definition – The Materiel Solution Analysis & AoA/Valuation Phase 6.1 Introduction If the Needs and Opportunities phase leads to a positive Materiel Development Decision, the project enters into the DoD Materiel Solution Analysis & Analysis of Alternatives phase. Section 4 of DoDI 5000.02 Enclosure 2 summarizes the system acquisition guidelines for this phase. Its primary objective is to satisfy the criteria for passing Milestone A. These criteria include performing analysis of at least two alternative Materiel Solution approaches, and providing evidence that at least one alternative is feasible to pursue, at least through the next phase of Technology Development and documentation of the systems Capability Development approach. Figure 6-1. Overview of the ICM – MSA and AoA Highlighted 6.2 Phase Key Elements Table 6-1 identifies the key phase elements in the Materiel Solution Analysis; Analysis of Alternatives phase. Each element will be elaborated in a section below. 106753919 - 62 - Version Date: 12/31/08 Chapter 6: Stage I: Incremental Definition – The Materiel Solution Analysis & AoA/Valuation Phase Version 0.5 Table 6-1 Materiel Solution Analysis; Analysis of Alternatives (Valuation) Goals & Objectives Entry Conditions Inputs Steps (concurrent) Exit Conditions Work Products Pitfalls Analysis of Alternatives with FED, detailed business case Key risks resolved or covered by risk mitigation plans Key infrastructure selected or underway Technology Development Strategy (TDS) Next-phase plans SCSH commitment to way forward Inadequate phase budget and schedule Neglecting SCSHs, particularly external interoperators Neglecting alternatives Easiest-first approaches Prototyping shortfalls, especially in scalability and off nominal cases Delays in FED, business-case preparation Shortfalls in FEDs, SCSH MOAs without risk mitigation plans Shortfalls in CP strategy Inadequate nextphase plans, budget, schedule CP = Competitive Prototyping FED = Feasibility Evidence Description IPT = Integrated Product Team MOE = Measures of Effectiveness MOA = Memorandum of Agreement OpCon = Operational Concept RAA = Responsibility, Authority, Accountability SCSH = Success-Critical Stakeholder TDS = Technology Development Strategy 106753919 Needs and Opportunities phase exit conditions Well-staffed and resourced IPTs Key risk and infrastructure plans Alternatives analysis or CP preparation plans: criteria, scenarios, testbeds for needs satisfaction, critical technology maturity, life cycle costs, manufacturing feasibility Needs and Opportunities phase work products Alternatives analysis/CP & infrastructure determination plans, staff, resources SCSH support for phase Elaborate IPT structure, RAAs, plans, and staffing and/or CP strategy/plans Establish crossIPT work rhythm and coordination Refine and execute phase and CP plans; monitor progress versus plans Monitor changes in needs, opportunities, and risks Adapt plans to needed change Select CPs; evaluate results Develop and validate TDS - 63 - Validated TDS: OpCon, prototypes, requirements, architecture, system and increment plans, feasibility/business -case MOE evidence or risk plans Key infrastructure selected or underway Detailed nextphase plans, including CP strategy, staffing, Cost and Software Data Reporting Committed to by SCSHs Validated TDS Prototyping of high-risk elements, system structure, and behavior Key support and infrastructu re selections for next phase Detailed next-phase plans SCSH Commitme nt MOAs Version Date: 12/31/08 Chapter 6: Stage I: Incremental Definition – Materiel Solution Analysis & AoA/Valuation Phase 6.3 Version 0.5 Goals and Objectives 6.3.1 Analysis of Alternatives with Feasibility Evidence Description, Detailed Business Case 6.3.2 Key Risks Resolved or Covered by Risk Mitigation Plans 6.3.3 Key Infrastructure Selected or Underway 6.3.4 Technology Development Strategy 6.3.5 Next-Phase Plans 6.3.6 Success-Critical Stakeholder Commitment to Way Forward 6.4 Entry Conditions 6.4.1 Needs and Opportunities Phase Exit Conditions 6.4.2 Well-Staffed and Resourced Integrated Product Teams 6.4.3 Key Risk and Infrastructure Plans 6.4.4 Alternatives Analysis or Competitive Prototyping Preparation Plans 6.5 Inputs 6.5.1 Needs and Opportunities Phase Work Products 6.5.2 Alternatives Analysis / Competitive Prototyping and Infrastructure Determination Plans, Staff, Resources 6.5.3 Success-Critical Stakeholder Support for Phase 6.6 Steps (Concurrent) 6.6.1 106753919 Elaborate Integrated Product Team Structure; Responsibility, Authority, Accountabilities; Plans, and Staffing and/or Competitive Prototyping Strategy/Plans - 64 - Version Date: 12/31/08 Chapter 6: Stage I: Incremental Definition – Materiel Solution Analysis & AoA/Valuation Phase Version 0.5 6.6.2 Establish Cross-Integrated Product Team Work Rhythm and Coordination 6.6.3 Refine and Execute Phase and Competitive Prototyping Plans; Monitor Progress Versus Plans 6.6.4 Monitor Changes in Needs, Opportunities, and Risks 6.6.5 Adapt Plans to Needed Change 6.6.6 Select Competitive Prototypes; Evaluate Results 6.6.7 Develop and Validate Technology Development Strategy 6.7 Exit Conditions 6.7.1 Validated Technology Development Strategy 6.7.2 Key Infrastructure Selected or Underway 6.7.3 Detailed Next-Phase Plans, Including Competitive Prototyping Strategy, Staffing, Cost and Software Data Reporting 6.7.4 Committed to by Success-Critical Stakeholders 6.8 Work Products 6.8.1 Validated Technology Development Strategy 6.8.2 Prototyping of High-Risk Elements, System Structure, and Behavior 6.8.3 Key Support and Infrastructure Selections for Next Phase 6.8.4 Detailed Next-Phase Plans 6.8.5 Success-Critical Stakeholder Commitment Memoranda of Agreement 6.9 Pitfalls 6.9.1 106753919 Inadequate Phase Budget and Schedule - 65 - Version Date: 12/31/08 Chapter 6: Stage I: Incremental Definition – Materiel Solution Analysis & AoA/Valuation Phase Version 0.5 6.9.2 Neglecting Success-Critical Stakeholders, Particularly External Interoperators 6.9.3 Neglecting Alternatives 6.9.4 Easiest-First Approaches 6.9.5 Prototyping Shortfalls, Especially in Scalability and Off-Nominal Cases 6.9.6 Delays in Feasibility Evidence Description, Business-Case Preparation 6.9.7 Shortfalls in Feasibility Evidence Descriptions, Success-Critical Stakeholder Memoranda of Agreement Without Risk Mitigation Plans 6.9.8 Shortfalls in Competitive Prototyping strategy 6.9.9 Inadequate Next-Phase Plans, Budget, Schedule 106753919 - 66 - Version Date: 12/31/08 Chapter 7: Stage I: Incremental Definition –The Technology Development & CDD/Foundations Phase Version 0.5 7Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase 7.1 Introduction The objective of the Technology Development and Capability Development Document phase is to ensure that the proposed acquisition program has fully prepared its technical and management foundations for passing Milestone B, initiating an acquisition program, and entering the Engineering and Manufacturing Development phase. These foundations include not only prototypes and other evidence of technology maturity, but also a fully developed system architecture and associated feasibility evidence sufficient to pass FED-based Preliminary Design Review (now scheduled before Milestone B in DoDI 5000.02), along with a fully elaborated Capability Development Document, with validated plans and feasibility evidence for cost and schedule estimates. This phase begins at the Anchor Point (AP) Foundations Commitment Review (FCR), and the passage of DoD Milestone A, as seen in Figure 7-1. 106753919 - 67 - Version Date: 12/31/08 Chapter 7: Stage I: Incremental Definition –The Technology Development & CDD/Foundations Phase Version 0.5 Figure 7-1. Overview of the ICM – Technology Development and CDD Highlighted Although the title of Section 5 of DoDI 5000.02’s Enclosure 2 is “Technology Development Phase,” its content includes the full set of technical and management foundations necessary for a successful Engineering and Manufacturing Development phase. Figure 7-2 shows the results of a root cause analysis of negative findings on over 50 DoD program assessments, which identified a number of risk sources that are not generally covered by the development and evaluation of technology. Overfocus on technology development without having good management processes, acquisition practices, requirements processes, and shortfalls in staffing, organizing, and contracting will lead to similar negative program findings. 106753919 - 68 - Version Date: 12/31/08 Chapter 7: Stage I: Incremental Definition –The Technology Development & CDD/Foundations Phase Version 0.5 Figure 7-2. Categorization of DoD Program Assessment Negative Findings Some of the root causes, such as verification and validation (V&V) shortfalls, competing priorities, lack of appropriate staff, and program realism, have been root causes of prototyping and technology maturity shortfalls, but causes of other shortfalls as well. There are good ways to address these additional risk sources within the scope of the new DoDI 5000.02 by placing stronger emphasis on such documents as the Technology Development Strategy and Capability Development Document, and on providing evidence of program achievability as well as technology maturity at Milestone B. As discussed below, the ICM provides process support for the concurrent development and validation of the risk-driven set of technology and management foundations. 7.2 Phase Key Elements Table 7-1 identifies the key phase elements in the Technology Development and Capabilities Development Document phase. Each element will be elaborated in a section below. 106753919 - 69 - Version Date: 12/31/08 Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase Version 0.5 Table 7-1. Technology Development and Capability Development Document: TD-CDD (Foundations) Goals & Objectives Entry Conditions Inputs Steps (concurrent) Exit Conditions Work Products Pitfalls Ensure Validated CDD Complete, Inadequate phase technology complete for validated budgets, readiness for increment 1 CDD for schedules mission needs increment 1 Neglecting Later features Establish prioritized and Validated SCSHs baseline EMD initially allocated acquisition Prototyping increment to later increments, strategy shortfalls content, budgets, including and Overemphasis on and schedules production, baseline technology: also sustainment plans, content for Prepare for, need refined ready technology later execute CP requirements, and infrastructure, increments, strategy validated designs, FED production Develop and cost estimates, Realistic risk Feasibility validate CDD: manufacturing mitigation plans evidence, OpCon, processes, infor remaining risks realistic acquisition place risk strategy, feature EMD and infrastructure, mitigation prioritization and production acquisition plans for allocation to acquisition management, remaining increments, strategy staff capabilities risks preliminary and Committed to by SCSH detailed design, SCSH SCSHs commitments development, Commitme Inadequate teamintegration and nt MOAs building test, feasibility Unprioritized evidence features Obtain decision Inadequate nextcommitments for phase budgets, EMD increment schedules 1 AoA = Analysis of Alternatives CDD = Capability Development Document CP = Competitive Prototyping EMD = Engineering and Manufacturing Development FED = Feasibility Evidence Description KPP = Key Performance Parameter MOA = Memorandum of Agreement MSA = Materiel Solution Analysis OpCon = Operational Concept SCSH = Success-Critical Stakeholder TD = Technology Development TDS = Technology Development Strategy Full TD-CDD with missionready technology, infrastructure, FED, life cycle and increment plans Risks resolved or covered by risk plans Worked-out supplier and strategicpartner participation plans and MOAs SCSH commitment to life-cycle plans and approach 106753919 MSA/AoA phase exit conditions Clear scoping of recommended -alternative objectives, constraints, and priorities Desired and acceptable KPP levels Approved TDS: Technology development, CP, risk resolution and infrastructure plans, staff, resources MSA/AoA phase work products Recommende d-alternative objectives, constraints, and priorities Desired and acceptable KPP levels Approved TDS Necessary staffing, budgets, schedules, info sources - 70 - Version Date: 12/31/08 Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase Version 0.5 7.3 Goals and Objectives 7.3.1 Full Technology Development and Capabilities Development Document with Mission-Ready Technology, Infrastructure, Feasibility Evidence Description, Life Cycle and Increment Plans 7.3.2 Risks Resolved or Covered by Risk Plans 7.3.3 Worked-Out Supplier and Strategic-Partner Participation Plans and Memoranda of Agreement 7.3.4 Success-Critical Stakeholder Commitment to Life-Cycle Plans and Approach 7.4 Entry Conditions 7.4.1 Materiel Solution Analysis / Analysis of Alternatives Phase Exit Conditions 7.4.2 Clear Scoping of Recommended-Alternative Objectives, Constraints, and Priorities 7.4.3 Desired and Acceptable Key Performance Parameter Levels 7.4.4 Approved Technology Development Strategy 7.5 Inputs 7.5.1 Materiel Solution Analysis / Analysis of Alternatives Phase Work Products 7.5.2 Recommended-Alternative Objectives, Constraints, and Priorities 7.5.3 Desired and Acceptable Key Performance Parameter Levels 7.5.4 Approved Technology Development Strategy 7.5.5 Necessary Staffing, Budgets, Schedules, Info Sources 7.6 106753919 Steps (Concurrent) - 71 - Version Date: 12/31/08 Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase Version 0.5 7.6.1 Ensure Technology Readiness for Mission Needs 7.6.2 Establish Baseline Engineering and Manufacturing Development Increment Content, Budgets, and Schedules 7.6.3 Prepare For and Execute Competitive Prototyping Strategy 7.6.4 Develop and Validate Capability Development Development 7.6.5 Obtain Decision Commitments for Engineering and Manufacturing Development Increment 1 7.7 Exit Conditions 7.7.1 Validated Capability Development Document Complete for Increment 1 7.7.2 Later Features Prioritized and Initially Allocated to Later Increments, Including Production, Sustainment Plans, Ready Technology and Infrastructure, Feasibility Evidence Document 7.7.3 Realistic Risk Mitigation Plans for Remaining Risks 7.7.4 Engineering and Manufacturing Development and Production Acquisition Strategy 7.7.5 Committed to by Success-Critical Stakeholders 7.8 Work Products 7.8.1 Complete, Validated Capability Development Document for Increment 1 7.8.2 Validated Acquisition Strategy and Baseline Content for Later Increments, Production 7.8.3 Feasibility Evidence, Realistic Risk Mitigation Plans for Remaining Risks 7.8.4 Success-Critical Stakeholder Commitment Memoranda of Agreement 106753919 - 72 - Version Date: 12/31/08 Chapter 7: Stage I: Incremental Definition – The Technology Development & CDD/Foundations Phase Version 0.5 7.9 Pitfalls 7.9.1 Inadequate Phase Budgets and Schedules 7.9.2 Neglecting Success-Critical Stakeholders 7.9.3 Prototyping Shortfalls 7.9.4 Overemphasis on Technology 7.9.5 Success-Critical Stakeholders Commitments 7.9.6 Inadequate Team-Building 7.9.7 Unprioritized Features 7.9.8 Inadequate Next-Phase Plans, Budget, Schedule 106753919 - 73 - Version Date: 12/31/08 Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V Version 0.5 8Chapter 8: Stage II: Incremental Developments – Engineering and Manufacturing Development and Beyond 8.1 Introduction As shown in the overall DoD-ICM life cycle diagram in Figure 8-1, Stage II covers not only Engineering and Manufacturing Development (EMD), but also how EMD increments are synchronized with activities beyond such as Production & Deployment and Operations & Maintenance. As discussed in the Executive Summary and Chapter 1 on ICM Principles, current and future trends toward emergent requirements and rapid change make incremental, evolutionary development the most cost-effective approach for most current and future DoD systems. Figure 8-1. Overview of the Generic Incremental Commitment Life Cycle Process – Stage II Highlighted 106753919 - 74 - Version Date: 12/31/08 Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V Version 0.5 DoD mission-critical systems must also have high levels of security, safety, reliability, availability, and maintainability, making agile development methods less viable. However, they may have less-critical components for which agile methods may be preferable. This Chapter will focus on Major Defense Acquisition Programs (MDAPs), but will include a section on special cases such as agile methods for less-critical, software-intensive systems of subsystems. Other special cases will include ICM variants for systems of systems, and for systems in which several software increments may fit within a longer hardware increment. Next, this chapter will continue with a more detailed discussion than in the Executive Summary of the overall structure of and motivation for the ICM Stage II process. It will then elaborate a representative Major Defense Acquisition Program (MDAP) increment in the previous format of Goals & Objectives, Preconditions, Inputs, Steps, Postconditions, Work Products, and Pitfalls. Finally, this chapter will address the special cases discussed above. 8.1.1 ICM Stage II Structure and Motivation Figure 8-2 shows a single increment of the ICM Stage II process. It assumes that the project has developed in Stage I and continues to evolve: A best-effort definition of the system’s long-range capability; An incremental sequence of prioritized capabilities culminating in the long-range capability; A Feasibility Rationale providing sufficient evidence that the system architecture will support the incremental capabilities, that each increment can be developed within its available budget and schedule, and that the series of increments create a satisfactory return on investment for the organization and mutually satisfactory outcomes for the successcritical stakeholders. 106753919 - 75 - Version Date: 12/31/08 Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V Version 0.5 Figure 8-2. ICM Stage II: Increment Activities As seen in Figure 8-2, the model is organized to simultaneously address the conflicting 21st century DoD challenges of rapid change and high assurance of dependability. It also addresses the need for rapid fielding of incremental capabilities with a minimum of rework, and the other major 21st DoD century trends involving integration of systems and software engineering, COTS components, legacy systems, net-centric services, and systems of systems.. Rapid change in DoD systems comes in two primary forms. One is relatively predictable change that can be handled by the plan-driven Parnas strategy of encapsulating sources of change within modules, so that the effects of changes are largely confined to individual modules. The other is relatively unpredictable change that may appear simple (such as adding a “cancel” or “undo” capability to the software or adding system payload components to the hardware), but often requires a great deal of agile adaptability to rebaseline the architecture and incremental capabilities into a feasible solution set. The need to deliver high-assurance incremental capabilities on short fixed schedules means that each increment needs to be kept as stable as possible. This is particularly the case for very large systems of systems with deep supplier hierarchies (often 6 to 10 levels), in which a high level of rebaselining traffic can easily lead to chaos. In keeping with the use of the ICM as a risk-driven process model generator, the risks of destabilizing the development process make this portion of the project into a build-to-specification approach in which unforeseeable changes are deferred. The need for high assurance of each increment also makes it cost-effective to invest in a team of appropriately skilled personnel to continuously verify and validate the increment as it is being developed. 106753919 - 76 - Version Date: 12/31/08 Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V Version 0.5 However, “deferring the change traffic” does not imply deferring its change impact analysis, change negotiation, and rebaselining until the beginning of the next increment. With a single development team and rapid rates of change, this would require a team optimized to develop to stable plans and specifications to spend much of the next increment’s scarce calendar time performing tasks much better suited to agile teams. Thus, it is important not to disband the systems engineering team at Milestone B, but to keep a critical mass of agile systems engineering talent on the project to handle the change traffic and rebaseline the plan, specifications, and feasibility rationales for the remaining increments. 8.1.2 Agile Rebaselining and the C2ISR Metaphor The appropriate metaphor for these tasks is not a build-to-specification metaphor or a purchasing-agent metaphor but an adaptive “command-control-intelligence-surveillancereconnaissance” (C2ISR) metaphor. It involves an agile team performing the first three activities of the C2ISR “Observe, Orient, Decide, Act” (OODA) loop for the next increments, while the plan-driven development team is performing the “Act” activity for the current increment. “Observing” involves monitoring changes in relevant technology and COTS products, in the competitive marketplace, in external interoperating systems, and in the environment; and monitoring progress on the current increment to identify slowdowns and likely scope deferrals. “Orienting” involves performing change impact analyses, risk analyses, and tradeoff analyses to assess candidate rebaselining options for the upcoming increments. “Deciding” involves stakeholder renegotiation of the content of upcoming increments, architecture rebaselining, and the degree of COTS upgrading to be done to prepare for the next increment. It also involves updating the future increments’ Feasibility Rationales to ensure that their renegotiated scopes and solutions can be achieved within their budgets and schedules and without compromising performance. A successful rebaseline means that the plan-driven development team can hit the ground running at the beginning of the “Act” phase of developing the next increment, and the agile team can hit the ground running on rebaselining definitions of the increments beyond. Figure 8-3 shows how this three-team cycle (lean, plan-driven, stabilized developers; thorough V&Vers; agile, proactive rebaseliners) plays out from one increment to the next. As much as possible, usage feedback from the previous increment (e.g., Operations and Support1) is not allowed to destabilize the current increment (e.g., EMD, PDR2), but is fed into the definition of the upcoming development increment (e.g., TD, CDD, LCSP3). Of course, some level of mission-critical updates will need to be fed into the current increment, but only when the risk of not doing so is greater that the risk of destabilizing the current increment. Stage I: Incremental Definition | Stage II: Incremental Development, Production & Ops. 106753919 - 77 - Version Date: 12/31/08 Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V Version 0.5 Figure 8-3. DoD Version of ICM for Evolutionary Development Again, in keeping with the use of the spiral model as a risk-driven process model generator, the risks of not adequately and rapidly adapting the next increments to address the sources of needed change mean that this portion of the project is not a build-to-specification waterfall process. Instead, it is a risk-driven set of concurrent prototyping, analysis, and stakeholder renegotiation activities leading to a best-possible redefinition of the plans and specifications to be used by the stabilized development team for the next increment. 8.2 Phase Key Elements Table 8-1 identifies the key phase elements in the Engineering and Manufacturing Developmentn phase. Each element is elaborated in a section below. 106753919 - 78 - Version Date: 12/31/08 Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V Version 0.5 Table 8-1. Engineering and Manufacturing Developmentn (Developmentn) Goals & Objectives Entry Conditions Inputs Steps (concurrent) Exit Conditions Work Products Develop, V&V, and transition the nth increment of capability Deliver on schedule, defer low-priority features if necessary Prepare rebaselined specifications, plans, FED for incrementn+1 Execute next phase of manufacturing plans Acceptable responses to change requests SCSH commitment to life-cycle plans and approach TD-CDDn exit conditions Stabilized and prioritized requirements, specifications, and plans Adequate staffing; funding of development, rebaselining, and V&V teams; and manufacturing capabilities TD-CDDn work products Incrementn development and V&V plans IPT, risk resolution, infrastructure, plans, staff, resources Change traffic from previousincrement users, management, technology, and competition Stabilized build-tospecifications development of incrementn Continuous V&V of build-tospecifications artifacts Next-increments rebaselining, FEDs based on change traffic inputs Development progress monitoring and scope adjustment Incrementn installation, operational test, training, and acceptance Incrementn manufacturing and test preparations Accepted incrementn operational capability Associated DOTMLPF capabilities Satisfactory disposition of change traffic Validated, rebaselined nextincrements CDDs Incrementn manufacturing and test plans and preparations Committed to by SCSHs Accepted incrementn operational capability Associated DOTMLPF capabilities Validated, rebaselined nextincrements CDDs Incrementn manufacturin g and test plans and preparations SCSH Commitment MOAs Pitfalls Inadequate phase budgets, schedules Neglecting SCSHs Destabilizing incrementn development Inadequate development monitoring and rescoping Inadequate test and transition plans and preparations Inadequate changesource monitoring and response Unvalidated and unprioritized nextincrement capabilities Weak manufacturing process and quality controls Inadequate nextphase budgets, schedules CDD = Capability Development Document DOTMLPF = Doctrine, Organization Training, Materiel, Leadership and education, Personnel, Facilities FED = Feasibility Evidence Description IPT = Integrated Product Team MDD = Materiel Development Decision MOA = Memorandum of Agreement OpCon = Operational Concept SCSH = Success-Critical Stakeholder TD = Technology Development V&V = Verification and Validation 106753919 - 79 - Version Date: 12/31/08 Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V 8.3 Version 0.5 Goals and Objectives 8.3.1 Develop, Verification and Validation, and Transition the nth Increment of Capability 8.3.2 Deliver on Schedule, Defer Low-Priority Features if Necessary 8.3.3 Prepare Rebaselined Specifications, Plans, Feasibility Evidence Document for Incrementn+1 8.3.4 Acceptable Responses to Change Requests 8.3.5 Success-Critical Stakeholder Commitment to Life-Cycle Plans and Approach 8.4 Entry Conditions 8.4.1 Technology Development-Capability Development Documentn Exit Conditions 8.4.2 Stabilized and Prioritized Requirements, Specifications, and Plans 8.4.3 Adequate Staffing; Funding of Development, Rebaselining, and Verification and Validation Teams; and Manufacturing Capabilities 8.5 Inputs 8.5.1 Technology Development-Capability Development Documentn Work Products 8.5.2 Incrementn Development and Verification and Validation Plans 8.5.3 Integrated Product Team, Risk Resolution, Infrastructure, Plans, Staff, Resources 8.5.4 Change Traffic from Previous-Increment Users, Management, Technology, and Competition 8.6 106753919 Steps (Concurrent) - 80 - Version Date: 12/31/08 Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V Version 0.5 8.6.1 Stabilized Build-to-Specifications Development of Incrementn 8.6.2 Continuous Verification and Validation of Build-to-Specifications Artifacts 8.6.3 Next-Increments Rebaselining, Feasibility Evidence Descriptions Based on Change Traffic Inputs 8.6.4 Development Progress Monitoring and Scope Adjustment 8.6.5 Incrementn Installation, Operational Test, Training, and Acceptance 8.6.6 Incrementn Manufacturing and Test Preparations 8.7 Exit Conditions 8.7.1 Accepted Incrementn Operational Capability 8.7.2 Associated Doctrine, Organization Training, Materiel, Leadership and Education, Personnel, Facilities Capabilities 8.7.3 Satisfactory Disposition of Change Traffic 8.7.4 Validated, Rebaselined Next-Increments Capability Development Documents 8.7.5 Incrementn Manufacturing and Test Plans and Preparations 8.8 Work Products 8.8.1 Accepted Incrementn Operational Capability 8.8.2 Associated Doctrine, Organization Training, Materiel, Leadership and Education, Personnel, Facilities Capabilities 8.8.3 Validated, Rebaselined Next-Increments Capability Development Documents 8.8.4 Incrementn Manufacturing and Test Plans and Preparations 8.8.5 Success-Critical Stakeholder Commitment Memoranda of Agreement 106753919 - 81 - Version Date: 12/31/08 Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V 8.9 Version 0.5 Pitfalls 8.9.1 Inadequate Phase Budgets and Schedules 8.9.2 Neglecting Success-Critical Stakeholders 8.9.3 Destabilizing Incrementn Development 8.9.4 Inadequate Development Monitoring and Rescoping 8.9.5 Inadequate Test and Transition Plans and Preparations 8.9.6 Inadequate Change-Source Monitoring and Response 8.9.7 Unvalidated and Unprioritized Next-Increment Capabilities 8.9.8 Weak Manufacturing Process and Quality Controls 8.9.9 Inadequate Next-Phase Plans, Budget, Schedule 8.10 Special Cases The agile, systems of systems, and hardware-software increment synchronization special case discussions will go here. Figure 8-4. Example of SoSE Coordination Challenge Note: Chapter 9 may be merged into Chapter 8, or it may continue to be a separate chapter. 106753919 - 82 - Version Date: 12/31/08 Chapter 8: Stage II: Incremental Development – Concurrent Development, V&V Version 0.5 Acknowledgements We would like to thank our sponsors in the OUSD(AT&L) Systems and Software Engineering Directorate, Kristen Baldwin and Bruce Amato, for their strategic guidance of this effort. We have also benefited greatly from the technical discussions on the ICM and its application to DoD projects with Elliot Axelband (Rand/USC), Winsor Brown (USC), Judith Dahmann (Mitre), Edgar Dalrymple (US Army), Chuck Driessnack (SAIC/MDA), Marilynn Goo (Boeing), Gary Hafen (LMCO), Peter Hantos (Aerospace), Dan Ingold (USC), Sue Koolmanojwong (USC), Ray Madachy (NPS), Art Pyster (Stevens), Rick Selby (NGC), Bob Skalamera (OSD/Stevens), Rich Turner (Stevens), Gan Wang (BAE Systems), and the members of the NRC study on HumanSystem Integration in the System Development Process. We owe a special degree of thanks to Alfred Brown for an outstanding job of technical editing. 106753919 - 83 - Version Date: 12/31/08