Final Report:
Integrating Systems and Software Engineering Project
USC Center for Systems and Software Engineering
Barry Boehm, Jo Ann Lane
December 2007
Executive Summary
This project was performed for the Office of the Deputy Under Secretary of Defense for
Acquisition and Technology (ODUSD(A&T)) Software Engineering and System Assurance
organization, as a subcontract under the Data Analysis Center for Software (DACS) contract
SP0700-98-D-4000 with ITT-AES, between 1 July 2007 and 31 December 2007. It
addresses two of the key software issues identified in the DUSD (A&T)-hosted Defense
Software Strategy Summit in October 2006. These are software life cycle planning and
management, and the integration of software engineering and systems engineering. The
project has consisted of three primary tasks:
1. Evaluate DoD systems engineering lifecycle processes and standards against typical
software engineering processes and standards. Prepare a report that delineates any
deltas, potential issue areas, and specific recommendations for integrating systems and
software engineering.
2. Evaluate the Incremental Commitment Model (ICM) to determine application to
integration of systems engineering, software engineering, and system acquisition life
cycles for DoD acquisitions.
3. Evaluate relevant DoD guidance, education, and training and provide
recommendations to incorporate findings from tasks 1 and 2.
Its context focused both on current systems and future trends toward rapidly-evolving, netcentric distributed and mobile systems of systems which also need high assurance of being
always-on, while being adaptive and robust in the face of system stress and adversary action.
The study focused on three primary sources of DoD systems engineering lifecycle processes
and standards: Version 1.0 of the AT&L-Enterprise Development “Systems Engineering Plan
Preparation Guide”; Chapter 4, “Systems Engineering,” of the Defense Acquisition
Guidebook; and DoD Instruction 5000.2. It evaluated them against such typical software
engineering processes and standards as the Rational Unified Process, the risk-driven spiral
model, ISO 12207, and the ICM as described in the 2007 National Research Council Report,
“Human-System Integration in the System Development Process.”
IS&SE Final Report
31 December 2007
1
Principal Findings
The study found that the DAG Chapter 4 and the SEP Preparation Guide were significant
advances over previous DoD guidance such as MIL-STD-499. They were less fixed on
sequential requirements-first processes, and more accommodating of early and thorough
verification and validation of requirements before proceeding into development. They had
more explicit support of risk management, of achieving appropriate technology readiness
levels, and of addressing the full range of system stakeholders. They explicitly addressed
system of systems and family of systems considerations, and emphasized event-based rather
than schedule-based milestone commitment points.
However, both documents contain many residues from traditional approaches to systems
engineering that confine themselves to physical systems engineering and cause significant
difficulties in integration with effective software engineering and human factors engineering
practices. Their use of one-to-many “part-of” and “owned-by” relationships in decomposing
systems conflicts strongly with the many-to-many “used-by” and service-oriented system
decompositions natural to systems with distributed, mobile, and rapidly-evolving software
and human elements of systems. Their hardware-oriented approach to reliability, availability,
and maintainability leaves serious shortfalls of emphasis with respect to their software and
human factors counterparts. Their emphasis on event-based milestones is a step forward
from schedule-based milestones, but usually leads to decision events that are heavy on
PowerPoint charts and Unified Modeling Language (UML) diagrams and weak on evidence
that a system built to these diagrams would satisfy the system’s requirements and be
buildable within its budgets and schedules.
Most critically, this hardware emphasis is propagated into the system development’s
management structures, such as its work breakdown structures, project management
hierarchies, and earned value management systems. Particularly in a rapidly changing
defense environment, the software and human factors performers enabling adaptability to
change are unable to rapidly traverse the hardware-oriented management structures to
execute short observe-orient-decide-act OODA-loops for new courses of action. Large,
complex systems have experienced change-processing times involving contract changes
averaging over 140 workdays, placing DoD at a serious disadvantage with respect to more
agile adversaries.
At a two-day workshop in October 2007, drafts of the findings and a summary of the ICM
and some of its applications to date were presented and discussed. The ICM is organized into
two major stages: an Incremental Definition stage and an Incremental Development and
Operations stage. Its key milestones map onto the major DoD 5000 series of milestones,
with a stronger emphasis on reviewing evidence of feasibility rather than just reviewing
PowerPoint charts and UML diagrams; identifying shortfalls in feasibility evidence as risks;
and basing milestone commitment decisions on the key stakeholders’ willingness to be
accountable for and accept the risks of proceeding.
In the ICM Incremental System Definition stage, the three phases of Exploration, Valuation,
and Architecting involve concurrent development of the system’s operational concept,
requirements, architecture, plans, prototypes, and feasibility evidence in increasing levels of
risk-driven detail. In the Incremental Development and Operations stage, three teams operate
IS&SE Final Report
31 December 2007
2
concurrently during each incremental development phase. One team performs a stabilized
build-to-specification of the time-determined highest-priority capabilities in the current
increment’s prioritized specifications, and prepares it for operations. A second team
performs continuous verification and validation of the first team’s artifacts. A third team
handles the change traffic from operations and external events, and rebaselines the plans and
specifications for the next series of increments, so that the first team can hit the ground
running in developing the next increment.
Various diagrams showing multiple views of the ICM are provided in the main report.
However, two-dimensional diagrams of complex, situation-dependent processes are too easy
to misinterpret for safe practical use. It is more important to follow the principles underlying
the ICM. These are:
1. Commitment and accountability
2. Stakeholder satisficing
3. Incremental growth
4. Concurrent engineering
5. Iterative development
6. Risk-based activities, artifacts, and milestones.
At the Workshop, working groups were formed to address technical, management, and
complex-systems aspects of integrating systems and software engineering. Their feedback
improved the draft findings on existing DoD systems engineering guidance, and generally
affirmed the strengths of the ICM while identifying a number of further ICM issues that need
to be further addressed before proposing it as official DoD guidance. These issues and toplevel responses are included in the workshop summary provided in Appendix A. They
included

Further experience on synchronization of hardware and software increments

Further experience on synchronization with externally-evolving system components

Detailed activity guidelines and generally available example artifacts

Budgeting for incremental vs. full-scale commitment

Multiple-contract provisions for parallel development, V&V, and next-increment
systems engineering

More usage experience in application to systems of systems

Shortfalls in available educational materials and courses.
The working groups also identified a set of findings and recommendations in such areas as
acquisition policy and practices; education and training; and research. These were outside of
the immediate scope of this study, but would be essential parts of a long-range strategy to
fully realize the benefits of integrating systems and software engineering. They are also
IS&SE Final Report
31 December 2007
3
included in Appendix A. Examples include recommendations to develop education, training,
mentoring, career path management, and incentives to create more software-capable systems
engineers and systems-capable software engineers; to accelerate research in the integration of
systems and software engineering capabilities and in the scalability and extendibility of
current systems and software engineering capabilities; and in general to prototype and pilot
proposed process, measurement and other changes before establishing them as recommended
or required practices.
Principal Study Conclusions and Recommendations
Current DoD systems engineering guidance is considerably better than its MIL-STD-499
predecessor, but it still has major shortfalls in effectively integrating software and human
factors engineering. This is especially the case with respect to future trends toward netcentric systems of systems, more rapid change, and needs for higher assurance of scalable
performance and qualities.
Most of the shortfalls can be overcome when being applied by experts. Unfortunately, many
programs have shortages of experts, particularly for software, and literal application of some
of the guidance shortfalls can lead to trouble in integrating systems and software engineering.
The identification of a shortfall is meant to be an indicator of an issue to consider carefully
during systems engineering of a program, and an item to consider in future versions of the
guidelines, and not as a reason to avoid use of the guidelines.
The ICM and its underlying principles are not revolutionary. They represent best commercial
practice, and make it easier to determine what kind of process best fits a given project, and
easier to keep the process on track and adaptive to change. Just as importantly, they make it
harder to misinterpret the process in dangerous ways, harder to gloss over key practices,
harder to neglect key stakeholders and disciplines, and harder to avoid accountability for the
project’s commitments.
The ICM is only partially proven in DoD practice, and can also be easily undermined by
inappropriate overriding acquisition practices and inflexible, one-size-fits-all contracting
mechanisms and incentive structures. On the positive side, there are ways to reinterpret
obsolete acquisition practices and policies to obtain good project outcomes without having to
wait for long-range policy changes. And its principles and practices have been used on quite
a few successful programs, including most of the annual Crosstalk Top-5 software-intensive
systems programs. Some current programs are investigating its tailored use on pilot projects.
Given this situation, the study recommendations involve a mixed strategy of

Opportunistically applying the ICM principles and selected ICM practices to
supplement current projects

Identifying and performing pilot projects to tailor, apply, evaluate, and upgrade the
full set of candidate ICM processes and practices

Concurrently developing, applying, and evolving draft guidance documents for use in
practice and in education and training of current and future DoD system developers
and managers.
IS&SE Final Report
31 December 2007
4
The ICM principles of commitment and accountability, stakeholder satisficing, incremental
growth, concurrent engineering, iterative development, and risk-based activities, artifacts, and
milestones should be reviewed as opportunities for tailoring to current projects, along with
selected ICM practices that can supplement current practices. Examples are:

Adding an evidence-based Feasibility Rationale to the content of current reviews such
as system and software requirements and design reviews

Stabilizing development increments by diverting most change traffic to a concurrent
systems engineering effort to incorporate the changes into future increment baselines

Performing continuous verification and validation, using such capabilities as an early
system integration facility, surrogate models and simulations, and an executing
architecture skeleton.
Some DoD organizations are coming forward as candidates to tailor, apply, evaluate, and
upgrade the full set of candidate ICM processes and practices on a pilot project representing
their future set of system engineering and development challenges. The study recommends
capitalizing on such opportunities, capturing experience data on their performance, and using
them as an experience base for determining which practices work best on which classes of
projects. This information would then provide a sound basis for developing a next generation
of acquisition and systems engineering practices better suited to the challenging needs of
complex, rapidly evolving future DoD systems.
IS&SE Final Report
31 December 2007
5