University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
System of Systems Engineering
Cost Modeling:
What Makes It Different from
Traditional Systems Engineering
Cost Modeling
Jo Ann Lane
USC CSSE
jolane@usc.edu
COCOMO Forum
October 2007
Ricardo Valerdi
MIT
rvalerdi@mit.edu
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Overview
• Overview of SoSs and SoSE
• Summary of key comparative research and pilot
studies
• Comparison of traditional SE and SoSE cost
models
• Conclusions
October 2007
©USC-CSSE
2
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Relationships between Traditional
Systems, SoSs, and Complex Systems
[Kuras and White, INCOSE, 2005]
October 2007
©USC-CSSE
3
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
What is a “System of Systems”?
• Very large systems developed by creating a framework or
architecture to integrate component systems
• SoS component systems independently developed and managed
– New or existing systems in various stages of development/evolution
– Have their own purpose
– Can dynamically come and go from SoS
• SoS exhibits emergent behavior not otherwise achievable by
component systems
• Typical domains
– Business: Enterprise-wide and cross-enterprise integration to support core
business enterprise operations across functional and geographical areas
– Military: Dynamic communications infrastructure to support operations in a
constantly changing, sometimes adversarial, environment
Based on Mark Maier’s SoS definition [Maier, 1998]
For more on SoS definitions, see [Lane and Valerdi, 2007]
October 2007
©USC-CSSE
4
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
TSE, SOSE, and Related Industry Standards
• Current standards for TSE processes
– ISO/IEC Standard 15288 – describes system processes from
system conception through system retirement
– ANSI/EIA Standard 632 – detailed processes for
conceptualization, development and transition to operation
(subset of ISO/IEC 15288)
– SEI’s Capability Maturity Model Integrated – framework for
defining an organization’s standard processes and procedures
– Defense Acquisition Guidebook (DAG) – references above as
examples of best practices, but defines own set of processes
• Guidelines for SoSE
– SoS SE Draft Guidebook (extension to DAG)
October 2007
©USC-CSSE
5
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
SoSE Compared to TSE Activities
•
•
SoSs have their own characteristics and associated challenges that are
different from traditional systems and large, complex systems
Reported areas of difference
–
–
–
–
–
•
Architecting
Prototypes/experimentation/tradeoffs
Scope of SoS
SoS performance
Maintenance and evolution
Key challenges for DoD SoSE
–
–
–
–
–
–
–
–
Business model and incentives to encourage working together (SoS level)
Doing the necessary tradeoffs at the SoS level
Human-system integration
Commonality of data, architecture, and business strategies (SoS level)
Removing multiple decision making layers
Requiring accountability at the enterprise level
Evolution management
Maturity of technology
October 2007
©USC-CSSE
6
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Recent Research Findings*
• Many types of SoS
• SoS Engineering Teams: Varying degrees of responsibility
and authority
• Incorporating many agile-like approaches to handle
– Multiple constituent systems
– Asynchronous activities and events
– Quickly take advantage of opportunities as they appear
• SoSE must
– Support multiple purposes and visions
– Requires significantly more negotiation
– Is content to satisfice rather than optimize
• SoSE activities map to traditional SE activities (e.g., DAG and
EIA 632), but take on a different focus and scope
* Based on USC CSSE SoSE cost model research,
MIT Systems Engineering Advancement Research Initiative, and OSD SoS SE pilot studies
October 2007
©USC-CSSE
7
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Relationship Among Core Elements of SoS SE
External Influences
Translating
capability
objectives
Understanding
systems &
relationships
(includes plans)
October 2007
Developing,
evolving and
maintaining
SoS
design/arch
Assessing
(actual)
performance
to capability
objectives
[assume these are fixed]
Block upgrade
process for SoS
Persistent
framework overlay
on systems in SoS
Monitoring
& assessing
changes
Addressing new
requirements
& options
Typically not the
role of the SE but
key to SoS
[architecture]
Orchestrating
upgrades
to SoS
©USC-CSSE
Large role of
external influences
8
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
SoSE Core Element Mapping to
COSOSIMO Sub-models
COSOSIMO
Translating
capability
objectives
Planning,
Requirements
Management,
and Architecting
(PRA)
Developing,
evolving and
maintaining
SoS
design/arch
Source Selection
and Supplier
Oversight (SO)
Addressing new
requirements
& options
Orchestrating
upgrades
to SoS
SoS Integration
and Testing
(I&T)
October 2007
Understanding
systems &
relationships
(includes plans)
Monitoring
& assessing
changes
©USC-CSSE
Assessing
(actual)
performance
to capability
objectives
9
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Comparison of System of Interest Focus Areas
Area
Traditional Systems
System of Systems
What to
engineer
Based on a set of
functional and
performance
requirements for the
system of interest
Based on a set of SoS capabilities that are then
translated into high level requirements for further analysis
A single capability can result in multiple requirements that
affect multiple constituent systems
View of
system-ofinterest
Clear system
boundaries
Interfaces
Systems that contribute to SoS capabilities and the
interrelationships between those systems
Architecture
Developed and
optimized to support
single purpose of
system
Net-centric, focused on information sharing
Does not address design details within constituent
systems, but rather the way the systems work together to
meet user needs
Sufficient versus optimized
Design
approach
Often top-down
Combined top-down and bottom-up, with focus on
–Existing assets (systems) that are within the SoS
–Opportunities within constituent system lifecycles
for changes
October 2007
©USC-CSSE
10
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Comparison of System of Interest Focus Areas (continued)
Area
Traditional Systems
System of Systems
Implementation
Contract-controlled, often
using an incremental,
evolutionary, or spiral
process
Focus on total system
SoS functionality implementation accomplished
through combination of negotiation, sometimes funded
by SoS or system owner, not always done via formal
agreements
Asynchronous and incremental due to lifecycles of
constituent systems
Primarily concerned with the implementation of SoS
functionality,
Monitors the evolution of constituent systems to
ensure that SoS is not adversely impacted, but not
typically involved in the implementation details
Testing
Traditional testing activities,
e.g., DT&E and OT&E
Attempt to leverage off of constituent system testing
Often impossible to test full-up SoS in a lab—often rely
on constituent system integration labs and operational
testing
Operationally, looking for how users use the system
and identifying emergent behavior for further analysis
October 2007
©USC-CSSE
11
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Areas of Emphasis in SE and SoS SE*
Area
TSE
SoS SE
Purpose
Development of a single system to meet
stakeholder requirements and defined
performance
Evolving new systems of systems
capability by leveraging synergies of
legacy systems and emerging capabilities
Systems
Architecture
Established early in the life cycle;
expectation set remains relatively stable
Dynamic adaptation as emergent needs
change
System
Interoperability
Interface requirements are defined and
implemented for the integration of
components in the system
Component systems can operate
independently of SoS in a useful manner;
protocols and standards are essential to
enable interoperable systems
System “ilities”
Reliability, maintainability, and
availability are typical concerns
Enhanced emphasis on ilities such as
flexibility, adaptability, and composability
Acquisition and
Management
Centralized acquisition and
management of the system
Component systems separately acquired
and continue to be managed and operated
as independent systems
Anticipation of
Needs
Concept phase activity to determine
system needs
Intense concept phase analysis followed
by continuous anticipation, aided by
ongoing experimentation
Funding
Single or homogenous stakeholder
group with stable cost/funding profile
and similar measures of success
Multiple heterogeneous stakeholder
groups with unstable cost/funding profile
and measures of success
October 2007
©USC-CSSE
* [Valerdi, et al., 2007]
12
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Engineering Cost and Schedule—What
Do Engineering Cost Models Look At?
• Engineering product characteristics
• Processes used to develop the product
• Skills and experience levels of the technical staff
responsible for development of product
• Size drivers used to compute nominal effort
• Cost drivers used to adjust nominal effort up or
down
October 2007
©USC-CSSE
13
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Comparison of Cost Model Parameters
Parameter Aspects
COSYSMO
COSOSIMO
Size drivers
# of system requirements
# of system interfaces
# operational scenarios
# algorithms
# of SoS requirements
# of SoS interface protocols
# of constituent systems
# of constituent system organizations
# operational scenarios
“Product” characteristics
Size/complexity
Requirements understanding
Architecture understanding
Level of service requirements
# of recursive levels in design
Migration complexity
Technology risk
#/ diversity of platforms/installations
Level of documentation
Size/complexity
Requirements understanding
Architecture understanding
Level of service requirements
Component system maturity and stability
Process characteristics
Process capability
Multi-site coordination
Tool support
Maturity of processes
Tool support
Cost/schedule compatibility
SoS risk resolution
People characteristics
Stakeholder team cohesion
Personnel/team capability
Personnel experience/continuity
Stakeholder team cohesion
SoS team capability
October 2007
©USC-CSSE
Component system readiness
14
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Impact of Differences to SE Cost Model
• Elements of SE that are not of major concern/driver of effort or can
be handled as a constant
–
–
–
–
–
–
Number of system algorithms
Number of recursion levels in the design
Migration complexity
Number/diversity of platforms/installations
Level of documentation
Multi-site coordination
• Elements of SoS SE not adequately addressed in SE cost model
– Analysis of capabilities to determine SoS and constituent system requirements
– Impact of negotiations required to develop architecture and enhancements
– Impact of number of constituent systems and organizations that own and
manage the constituent systems
– The asynchronous nature of constituent component changes and the
accommodation of partial increment implementations operationally
– Additional effort required to accommodate unplanned constituent component
changes
– Uncertainty with respect to funding sources and impact on required negotiations
October 2007
©USC-CSSE
15
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Conclusions to Date and Future Work
• Recent research efforts articulate some significant
differences between TSE and SoSE
– At a basic level, many of the activities are the same
– However, there are significant differences to the scope and
focus of the engineering activities
• Major focus of/impacts to SoSs
– Legacy systems and the independent control of them
– Focus on inter-relationships between relatively independent
components vs. integration of tightly coupled components
• Elements of SoS SE not adequately addressed in
existing traditional SE cost models
– COSOSIMO strives to address these SoS SE differences
– We need industry support to better understand and calibrate
these differences
October 2007
©USC-CSSE
16
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
Backup Charts
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
SoSE Compared to Traditional SE Activities:
Reported Differences
•
•
Architecting
– Architecting composability vs. decomposition (Meilich 2006)
– Net-friendly vs. hierarchical (Meilich 2006)
Prototypes/experimentation/tradeoffs
– Early tradeoffs/evaluations of alternatives (Finley 2006)
– Intense concept phase analysis followed by continuous
anticipation; aided by ongoing experimentation (USAF 2005)
– Modeling and simulation, in particular to better understand
“emergent behaviors” (Finley 2006)
– First order tradeoffs above the component systems level (e.g.,
optimization at the SoS level, instead of at the component system
level) (Garber 2006)
– Discovery and application of convergence protocols (USAF 2005)
October 2007
©USC-CSSE
18
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
SoSE Compared to Traditional SE Activities:
Reported Differences (continued)
•
•
Scope and performance
– Added “ilities” such as flexibility, adaptability, composability
(USAF 2005)
– Human as part of the SoS (Siel 2006, Meilich 2006, USAF 2005)
– Organizational scope defined at runtime instead of at system
development time (Meilich 2006)
– Dynamic reconfiguration of architecture as needs change (Meilich
2006)
Maintenance and evolution
– Component systems separately acquired and continue to be
managed as independent systems (USAF 2005)
October 2007
©USC-CSSE
19
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
SoSE Core Element Descriptions
• Translating capability objectives
– Developing a basic understanding of the expectations of the SoS and
the core requirements for meeting these expectations, independent of
the systems that comprise the SoS
• Understanding systems and relationships
– In a SoS, the focus is on the systems which contribute to SoS SE
capabilities and their interrelationships (as opposed to in a system,
the focus is on boundaries and interfaces)
• Assessing actual performance to capability objectives
– Establishing SoS metrics and methods for assessing performance and
conducting evaluations of actual performance using metrics and
methods
• Developing, evolving, and maintaining an SoS architecture/design
– Establishing and maintaining a persistent framework for addressing
the evolution of the SoS to meet user needs, including possible
changes in systems functionality, performance or interfaces
October 2007
©USC-CSSE
20
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
SoSE Core Element Descriptions
(continued)
• Monitoring and assessing changes
– Monitoring proposed or potential changes and assessing their
impacts to:
• Identify opportunities for enhanced functionality &
performance, and
• Preclude or mitigate problems for the SoS and constituent
systems (this may include negotiating with the constituent
system over how the change is made, in order to preclude
SoS impacts
• Addressing new requirements and options
– Reviewing, prioritizing, and determining which SoS
requirements to implement next
• Orchestrating upgrades to SoS
– Planning, facilitating, integrating, testing changes in systems to
meet SoS needs
October 2007
©USC-CSSE
21
University of Southern California
Center for Systems and Software Engineering
Massachusetts Institute of Technology
References
Dahmann, J. (2007); “Systems of Systems Challenges for Systems Engineering”, Systems and Software Technology
Conference, June 2007.
DiMario, Mike (2006); “System of Systems Characteristics and Interoperability in Joint Command Control”, Proceedings of the
2nd Annual System of Systems Engineering Conference
Electronic Industries Alliance (1999); EIA Standard 632: Processes for Engineering a System
Finley, James (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference
Garber, Vitalij (2006); “Keynote Presentation”, Proceedings of the 2nd Annual System of Systems Engineering Conference
INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP-2003-002-03
Krygiel, A. (1999); Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33
Kuras, M. L., White, B. E., Engineering Enterprises Using Complex-System Engineering, INCOSE Symposium 2005.
Lane, J., Valerdi, R., “Synthesizing System-of-Systems Concepts for Use in Cost Modeling,” Systems Engineering, Vol. 10,
No. 4, December 2007.
Maier, M. (1998); “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267-284)
Meilich, Abe (2006); “System of Systems Engineering (SoSE) and Architecture Challenges in a Net Centric Environment”,
Proceedings of the 2nd Annual System of Systems Engineering Conference
Pair, Major General Carlos (2006); “Keynote Presentation”, Proceedings of the 2nd Annual System of Systems Engineering
Conference
Proceedings of AFOSR SoSE Workshop, Sponsored by Purdue University, 17-18 May 2006
Proceedings of Society for Design and Process Science 9th World Conference on Integrated Design and Process Technology,
San Diego, CA, 25-30 June 2006
Siel, Carl (2006); “Keynote Presentation”, Proceedings of the 2nd Annual System of Systems Engineering Conference
Valerdi, R. (2005); Constructive Systems Engineering Cost Model. PhD. Dissertation, University of Southern California
Valerdi, R., Ross, A., Rhodes, D., “A Framework for Evolving System of Systems Engineering,” CrossTalk - The Journal of
Defense Software Engineering, October 2007.
United States Air Force Scientific Advisory Board (2005); Report on System-of-Systems Engineering for Air Force Capability
Development; Public Release SAB-TR-05-04
October 2007
©USC-CSSE
22