Final Report: Integrating Systems and Software Engineering Project USC Center for Systems and Software Engineering Barry Boehm, Jo Ann Lane December 2007 Executive Summary This project was performed for the Office of the Deputy Under Secretary of Defense for Acquisition and Technology (ODUSD(A&T)) Software Engineering and System Assurance organization, as a subcontract under the Data Analysis Center for Software (DACS) contract SP0700-98-D-4000 with ITT-AES, between 1 July 2007 and 31 December 2007. It addresses two of the key software issues identified in the DUSD (A&T)-hosted Defense Software Strategy Summit in October 2006. These are software life cycle planning and management, and the integration of software engineering and systems engineering. The project has consisted of three primary tasks: 1. Evaluate DoD systems engineering lifecycle processes and standards against typical software engineering processes and standards. Prepare a report that delineates any deltas, potential issue areas, and specific recommendations for integrating systems and software engineering. 2. Evaluate the Incremental Commitment Model (ICM) to determine application to integration of systems engineering, software engineering, and system acquisition life cycles for DoD acquisitions. 3. Evaluate relevant DoD guidance, education, and training and provide recommendations to incorporate findings from tasks 1 and 2. Its context focused both on current systems and future trends toward rapidly-evolving, netcentric distributed and mobile systems of systems which also need high assurance of being always-on, while being adaptive and robust in the face of system stress and adversary action. The study focused on three primary sources of DoD systems engineering lifecycle processes and standards: Version 1.0 of the AT&L-Enterprise Development “Systems Engineering Plan Preparation Guide”; Chapter 4, “Systems Engineering,” of the Defense Acquisition Guidebook; and DoD Instruction 5000.2. It evaluated them against such typical software engineering processes and standards as the Rational Unified Process, the risk-driven spiral model, ISO 12207, and the ICM as described in the 2007 National Research Council Report, “Human-System Integration in the System Development Process.” IS&SE Final Report 31 December 2007 1 Principal Findings The study found that the DAG Chapter 4 and the SEP Preparation Guide were significant advances over previous DoD guidance such as MIL-STD-499. They were less fixed on sequential requirements-first processes, and more accommodating of early and thorough verification and validation of requirements before proceeding into development. They had more explicit support of risk management, of achieving appropriate technology readiness levels, and of addressing the full range of system stakeholders. They explicitly addressed system of systems and family of systems considerations, and emphasized event-based rather than schedule-based milestone commitment points. However, both documents contain many residues from traditional approaches to systems engineering that confine themselves to physical systems engineering and cause significant difficulties in integration with effective software engineering and human factors engineering practices. Their use of one-to-many “part-of” and “owned-by” relationships in decomposing systems conflicts strongly with the many-to-many “used-by” and service-oriented system decompositions natural to systems with distributed, mobile, and rapidly-evolving software and human elements of systems. Their hardware-oriented approach to reliability, availability, and maintainability leaves serious shortfalls of emphasis with respect to their software and human factors counterparts. Their emphasis on event-based milestones is a step forward from schedule-based milestones, but usually leads to decision events that are heavy on PowerPoint charts and Unified Modeling Language (UML) diagrams and weak on evidence that a system built to these diagrams would satisfy the system’s requirements and be buildable within its budgets and schedules. Most critically, this hardware emphasis is propagated into the system development’s management structures, such as its work breakdown structures, project management hierarchies, and earned value management systems. Particularly in a rapidly changing defense environment, the software and human factors performers enabling adaptability to change are unable to rapidly traverse the hardware-oriented management structures to execute short observe-orient-decide-act OODA-loops for new courses of action. Large, complex systems have experienced change-processing times involving contract changes averaging over 140 workdays, placing DoD at a serious disadvantage with respect to more agile adversaries. At a two-day workshop in October 2007, drafts of the findings and a summary of the ICM and some of its applications to date were presented and discussed. The ICM is organized into two major stages: an Incremental Definition stage and an Incremental Development and Operations stage. Its key milestones map onto the major DoD 5000 series of milestones, with a stronger emphasis on reviewing evidence of feasibility rather than just reviewing PowerPoint charts and UML diagrams; identifying shortfalls in feasibility evidence as risks; and basing milestone commitment decisions on the key stakeholders’ willingness to be accountable for and accept the risks of proceeding. In the ICM Incremental System Definition stage, the three phases of Exploration, Valuation, and Architecting involve concurrent development of the system’s operational concept, requirements, architecture, plans, prototypes, and feasibility evidence in increasing levels of risk-driven detail. In the Incremental Development and Operations stage, three teams operate IS&SE Final Report 31 December 2007 2 concurrently during each incremental development phase. One team performs a stabilized build-to-specification of the time-determined highest-priority capabilities in the current increment’s prioritized specifications, and prepares it for operations. A second team performs continuous verification and validation of the first team’s artifacts. A third team handles the change traffic from operations and external events, and rebaselines the plans and specifications for the next series of increments, so that the first team can hit the ground running in developing the next increment. Various diagrams showing multiple views of the ICM are provided in the main report. However, two-dimensional diagrams of complex, situation-dependent processes are too easy to misinterpret for safe practical use. It is more important to follow the principles underlying the ICM. These are: 1. Commitment and accountability 2. Stakeholder satisficing 3. Incremental growth 4. Concurrent engineering 5. Iterative development 6. Risk-based activities, artifacts, and milestones. At the Workshop, working groups were formed to address technical, management, and complex-systems aspects of integrating systems and software engineering. Their feedback improved the draft findings on existing DoD systems engineering guidance, and generally affirmed the strengths of the ICM while identifying a number of further ICM issues that need to be further addressed before proposing it as official DoD guidance. These issues and toplevel responses are included in the workshop summary provided in Appendix A. They included Further experience on synchronization of hardware and software increments Further experience on synchronization with externally-evolving system components Detailed activity guidelines and generally available example artifacts Budgeting for incremental vs. full-scale commitment Multiple-contract provisions for parallel development, V&V, and next-increment systems engineering More usage experience in application to systems of systems Shortfalls in available educational materials and courses. The working groups also identified a set of findings and recommendations in such areas as acquisition policy and practices; education and training; and research. These were outside of the immediate scope of this study, but would be essential parts of a long-range strategy to fully realize the benefits of integrating systems and software engineering. They are also IS&SE Final Report 31 December 2007 3 included in Appendix A. Examples include recommendations to develop education, training, mentoring, career path management, and incentives to create more software-capable systems engineers and systems-capable software engineers; to accelerate research in the integration of systems and software engineering capabilities and in the scalability and extendibility of current systems and software engineering capabilities; and in general to prototype and pilot proposed process, measurement and other changes before establishing them as recommended or required practices. Principal Study Conclusions and Recommendations Current DoD systems engineering guidance is considerably better than its MIL-STD-499 predecessor, but it still has major shortfalls in effectively integrating software and human factors engineering. This is especially the case with respect to future trends toward netcentric systems of systems, more rapid change, and needs for higher assurance of scalable performance and qualities. Most of the shortfalls can be overcome when being applied by experts. Unfortunately, many programs have shortages of experts, particularly for software, and literal application of some of the guidance shortfalls can lead to trouble in integrating systems and software engineering. The identification of a shortfall is meant to be an indicator of an issue to consider carefully during systems engineering of a program, and an item to consider in future versions of the guidelines, and not as a reason to avoid use of the guidelines. The ICM and its underlying principles are not revolutionary. They represent best commercial practice, and make it easier to determine what kind of process best fits a given project, and easier to keep the process on track and adaptive to change. Just as importantly, they make it harder to misinterpret the process in dangerous ways, harder to gloss over key practices, harder to neglect key stakeholders and disciplines, and harder to avoid accountability for the project’s commitments. The ICM is only partially proven in DoD practice, and can also be easily undermined by inappropriate overriding acquisition practices and inflexible, one-size-fits-all contracting mechanisms and incentive structures. On the positive side, there are ways to reinterpret obsolete acquisition practices and policies to obtain good project outcomes without having to wait for long-range policy changes. And its principles and practices have been used on quite a few successful programs, including most of the annual Crosstalk Top-5 software-intensive systems programs. Some current programs are investigating its tailored use on pilot projects. Given this situation, the study recommendations involve a mixed strategy of Opportunistically applying the ICM principles and selected ICM practices to supplement current projects Identifying and performing pilot projects to tailor, apply, evaluate, and upgrade the full set of candidate ICM processes and practices Concurrently developing, applying, and evolving draft guidance documents for use in practice and in education and training of current and future DoD system developers and managers. IS&SE Final Report 31 December 2007 4 The ICM principles of commitment and accountability, stakeholder satisficing, incremental growth, concurrent engineering, iterative development, and risk-based activities, artifacts, and milestones should be reviewed as opportunities for tailoring to current projects, along with selected ICM practices that can supplement current practices. Examples are: Adding an evidence-based Feasibility Rationale to the content of current reviews such as system and software requirements and design reviews Stabilizing development increments by diverting most change traffic to a concurrent systems engineering effort to incorporate the changes into future increment baselines Performing continuous verification and validation, using such capabilities as an early system integration facility, surrogate models and simulations, and an executing architecture skeleton. Some DoD organizations are coming forward as candidates to tailor, apply, evaluate, and upgrade the full set of candidate ICM processes and practices on a pilot project representing their future set of system engineering and development challenges. The study recommends capitalizing on such opportunities, capturing experience data on their performance, and using them as an experience base for determining which practices work best on which classes of projects. This information would then provide a sound basis for developing a next generation of acquisition and systems engineering practices better suited to the challenging needs of complex, rapidly evolving future DoD systems. IS&SE Final Report 31 December 2007 5