Paper #062
Assessment of Process Modeling Tools to Support the
Analysis of System of Systems Engineering Activities
Jo Ann Lane, F. Stan Settles, and Barry Boehm
University of Southern California Center for Systems and Software Engineering
Los Angeles, CA 90089
{jolane, settles, boehm}@usc.edu
The goal of this approach is to build on
existing capabilities to produce new
capabilities not provided by the existing
systems. With this development approach,
system development processes to define the
new architecture, identify sources to either
supply or develop the required components,
and eventually integrate and test these high
level components are evolving and are being
referred to as SoS Engineering (SoSE).
Recent reports (DoD 2006, Northrup et al
2006) are indicating that SoSE activities are
considerably different from the more
classical systems engineering (SE) activities.
Other systems engineering experts believe
that there is nothing really different with
respect to systems engineering activities or
component-based engineering in the SoS
environment—that there are only differences
in scale and complexity (DoD 2006, Lane
2005a). However, most of these beliefs are
opinions based on observations, albeit from
professionals working in the SoSE arena,
and not substantiated by case study analyses
or data. Currently, research efforts are
underway at the University of Southern
California (USC) Center for Systems and
Software Engineering (CSSE) to identify
differences between SoSE and classical SE
and, if significant differences do exist with
respect to cost/effort, to capture these
differences in an SoSE cost estimation
model. This paper reports on a related
research effort to identify process modeling
tool(s) to support the SoSE-SE comparison.
Abstract
Many organizations are attempting to
provide new system capabilities through the
net-centric integration of existing systems
into systems of systems (SoS).
The
engineering activities used to architect and
develop these SoS are often referred to as
SoS Engineering (SoSE). Recent reports are
indicating that SoSE activities are
considerably different from classical
systems engineering (SE) activities. Other
systems engineering experts believe that
there is nothing really different with respect
to systems engineering activities or
component-based engineering in the SoS
environment—that there are only differences
in scale and complexity. To better
understand SoSE, studies are currently
underway to evaluate the differences
between classical SE and SoSE. This paper
summarizes process areas to be investigated
in the SE-SoSE comparison and then
analyzes and evaluates several types of
process modeling tools in order to identify a
set of tools that can be used to capture
classical
SE
and
SoSE
process
characteristics for further comparison.
Introduction
Many organizations are attempting to
provide new system capabilities through the
net-centric integration of existing systems.
1
The goal is to identify one or more process
modeling tools that will support the capture
and analysis of SoSE and classical SE
activities and facilitate comparison of the
two using information from as many as 40
projects.
This paper first provides the context for
this research by describing what is meant by
“SoS’ and “SoSE”. The following section
describes the process model tool capabilities
that are desired for the classical SE-SoSE
process comparison.
Next, this paper
presents an overview of the candidate
process modeling tools evaluated. A sample
SoSE project is used to facilitate the
evaluation of the process model tools and
the following section provides an overview
of that sample project. The analysis section
identifies the criteria used to evaluate each
candidate tool and presents the results of the
analysis. The last section presents the
process model tool evaluation conclusions.
managed, have their own purpose, and can
operate either on their own or within the
SoS.
In the business domain, an SoS is the
enterprise-wide or multiple enterprise
integration and sharing of core business
information
across
functional
and
geographical areas. In the military domain,
an SoS is a dynamic communications
infrastructure and a configurable set of
component systems to support operations in
a
constantly
changing,
sometimes
adversarial, environment. For some, an SoS
may be a multi-system architecture that is
planned up-front by a prime contractor or
lead system integrator. For others, an SoS is
an architecture that evolves over time, often
driven by organization needs, new
technologies appearing on the horizon, and
available budget and schedule.
The
evolutionary SoS architecture is more of a
network architecture that is reconfigured and
grows with needs and available resources.
In any case, users and nodes in the SoS
network may be either fixed or mobile.
Communications
between
component
systems in the SoS are often some
combination of standard and custom
protocols. Networks may tie together other
networks as well as nodes and users. SoS
component systems typically come and go
over time. As mentioned above, these
component systems can operate both within
the SoS framework and independent of this
framework.
In a general sense, it is
challenging to define clear boundaries of an
SoS because of its dynamic nature.
What is SoSE. With the SoS
development approach, system development
processes are evolving and are being
referred to as SoS Engineering (SoSE)
(SoSECE 2006, USAF SAB 2005). SoSE is
that set of engineering activities performed
to define the desired SoS-level capabilities,
develop the SoS-level architecture, identify
sources to either supply or develop the
Background
Key to this research activity is having a
general concept of what an SoS is and what
is SoSE. The following sections provide an
overview of each.
What is an SoS. A review of recent
publications (Carlock et al. 2006, Lane et al.
2005, and Wang et al. 2006) shows that the
term “system-of-systems” means many
things to many different people and
organizations. For the purposes of this
research, an SoS definition based on (Maier
1998) is used: an evolutionary net-centric
architecture that allows component systems
to exchange information and perform tasks
within the framework that they are not
capable of performing on their own outside
of the framework. This ability to perform
tasks that cannot be performed by any of the
component systems outside of the SoS
framework is often referred to as “emergent
behavior”.
In addition, the component
systems are typically independently
2
required SoS component systems, then
integrate and test these high level
components within the SoS architecture
framework, with the result being a
deployable SoS.
be more tied to the organization than to
differences between classical SE and SoSE.
The second major consideration is the
ability to capture aspects that might be
significantly different between SoSE and
classical SE activities.
From literature
research (Lane 2006b), several candidate
areas of differences have been identified.
These include:
 % of innovative or immature
technology
 Number of parallel engineering
activities
 Order in which engineering
activities/processes are performed
 % of activities focused on SoS
interfaces (internal and external)
 % of activities/effort focused on netcentric technology/issues
 % of activities/effort focused on
information/knowledge management
issues
 Impact of standards/lack of standards
 Impact of protocols/lack of standard
or convergent protocols
 Organizational management issues
related to multiple partners/suppliers
 Organizational management issues
related to distributed teams
 New engineering specialties required
to develop net-centric softwareintensive SoSs
 New contracting approaches needed
to support collaborative development
and rapid/continual change
 Competing business goals for
suppliers/vendors (e.g., engineering
for the good of the SoS as opposed to
engineering to optimize SoS
component systems)
 Lifecycle model differences.
These potential differences were used to
develop a set of criteria (described later in
the analysis section) for evaluating the
process modeling tools. The goal was to
Desired Process Model Tool
Capabilities
The goal of this research is to evaluate a
set of process modeling tools to determine
how well each one can support a
comparative analysis of classical SE
activities with SoSE activities. The first
consideration is the level of detail of process
information to be captured. Process models
can be viewed using Alistair Cockburn’s
hierarchy for use cases (Cockburn 2001). In
this hierarchy, very high level descriptions
are referred to as “sky-level”. For process
models, this corresponds to generic process
models used to describe classic models such
as waterfall, evolutionary, spiral, and the
“V” model. At the other extreme, referred
to as “underwater-level” in Cockburn’s
hierarchy, are the very detailed process
procedures that describe the process inputs,
entry criteria, responsible entities, steps, exit
criteria, and outputs. What is desired for the
SoSE-classical SE comparison is something
in between these two extremes, a level of
detail more akin to Cockburn’s “sea-level”
descriptions. The desired process model
tools should allow the modeller to specify
details below the high level phase or stage
descriptions,
but
not
require
the
specification of detailed process procedures.
This level of detail should provide
information at a level that allows
identification of activities that might be
different between classical SE projects and
SoSE projects without overwhelming project
personnel with excessive requests for
information. In addition, this level should
ensure that the analysis does not focus on
organizational details and culture that might
3
identify tools that would be able to capture
information at a level of detail that would
clearly highlight any significant differences
between classical SE and SoSE.
process mapping are described in (Breyfogle
2003). Process flowcharts are constructed
using terminal symbols, activity symbols,
decision symbols, and on-page and off-page
connectors, as illustrated in Figure 1. These
symbols are connected using directed
arrows, with the arrowhead showing the
direction of flow. Often these flows are
developed on a chart showing major
organizational entities responsible for the
activities (or processes and sub-processes)
and decisions. These are similar in nature to
Unified Modeling Language (UML)
Activity Diagrams described in (OMG
2007).
Overview of Candidate Process
Modeling Tools
There are a variety of process modeling
tools available in the marketplace that
support a variety of purposes: project
planning and analysis, development of
standard business processes, business
process re-engineering, and analysis of
activities for lean-Six Sigma improvements
to name a few. Because this evaluation is
being conducted to compare SoSE activities
with more classical SE activities, this
evaluation looked at the tools primarily
oriented towards a) project planning and
analysis, b) process documentation and
analysis tools to support lean-Six Sigma
initiatives or business process reengineering, and c) analysis tools to
understand influences on activities or tasks.
The tools selected for this evaluation
include:
 Process flowcharts that are often
used as part of Six Sigma
improvement activities to understand
current organizational processes

Org 1
Org 2
…
Org n
Start
Activity 1
Decision
Activity 3
Activity 4
Activity 2
…
End
Figure 1. Process Flowchart Format.
Microsoft (MS) Project, a project
planning, analysis, and tracking tool
that is very similar to other business
process modeling tools oriented
around a Work Breakdown Structure
(WBS) format
MS Project. MS Project is a project
management tool designed to help managers
organize and track tasks and resources
associated with a project or program. It is
primarily oriented towards managing
resources and schedules, given a set of tasks,
activities, or processes in a WBS type of
format with associated Gantt or network
diagrams. The Gantt format is shown in
Figure 2. Note that this tool was selected as
a representative of several business process
engineering tools such as Intelligent
System’s
ProcessEdge™
(Intelligent
Systems Technology Incorporated 2004) and

Vensim,
a
system
dynamics
modeling tool used to model
influences on process activities.
The following sections describe these
tools in more detail.
Process Flowcharts. The use of process
flowcharts and the associated standards and
conventions with respect to Six Sigma
4
Eclipse Process Framework (EPF) (EPF
2006), to name a few. At a high level, these
tools were very similar. However, the more
specific business process engineering tools
had additional capabilities to define process
artifacts and procedures which were deemed
too detailed for the classical SE-SoSE
comparison. Therefore, MS Project was
used as a reasonable representative of this
type of tool. Additional information on MS
Project can be found in (Microsoft 2003).
Figure 3. MS Project Format
Vensim. Vensim, described in (Ventana
Systems Incorporated 2006), is a visual
modeling tool that allows one to
conceptualize, document, simulate, analyze,
and optimize models of dynamic systems
and processes. Simulation models are built
from causal loop or stock and flow
diagrams, as shown in Figure 3.
Figure 3. Vensim Example (Ventana Systems Incorporated 2006).
5
Relationships among system variables
(or influences) are entered as causal
connections.
The model is analyzed
throughout the building process by looking
at the causes and uses of a variable and at
the loops involving the variable. Models
built using Vensim are also executable,
allowing the user to explore the behavior of
the model and conduct “what if” analyses.
includes the integration of housing,
transportation,
commissary,
medical
services, pharmacy, counseling services,
inventory management, and inmate trust
accounting. As part of these capabilities, the
system includes component systems for
photo imaging, inmate bar coding, and
fingerprint identification. In addition, this
SoS was designed as a seven-node system
geographically distributed across seven jail
sites. The system can operate with just a
single node or up to all seven nodes from
any jail site.
The sample SoS project was selected for
the evaluation because it exhibited several
key characteristics common to SoS
developments: the involvement of multiple
organizations, the development/tailoring of
multiple component systems in parallel, and
an emphasis on business process reengineering to support the SoS transition to
operations. In addition, it possesses all of
the SoS characteristics described in (Maier
1998). Table 1 summarizes the Project A
characteristics that map to the general SoS
characteristics described in (Maier 1998).
Overview of SoSE Project Used to
Evaluate Tools
To help analyze the process models tools
identified above, a sample SoS project was
selected. The goal is to evaluate how well
each tool captures the types of information
listed above for a given project. The project,
referred to as Project A, was a multi-year
effort, supported by multiple organizations,
to replace an existing jail inmate tracking
legacy system as well as to extend the
existing legacy system capabilities through
the integration of several other systems.
Project A can be viewed as an SoS since, in
additional to the inmate tracking capabilities
that replace the legacy system, it also
Table 1. Project A SoS Characteristics.
SoS Characteristic
Corresponding Project A Feature
Operational Independence of the Elements
All component systems can operate independent of the SoS
Managerial Independence of the Elements
All component systems are commercial-off-the-shelf components
managed by the respective vendors
Evolutionary Development
The SoS continues to evolve as capabilities are developed to
access the system from law enforcement vehicles in the field and
to integrate capabilities from other government systems
Emergent Behavior
Sample emergent behaviors: positive identification of people
booked into jail, continual tracking of inmate locations (e.g., jail
cell, court, medical), enhanced staff and inmate safety
Analysis of Tools
Initial Analysis. To conduct the tool
evaluation, the sample project’s SoSE
processes were described using each of the
tools under consideration.
During the
development of the project descriptions, the
tools were evaluated against a set of desired
capabilities.
Table 2 summarizes this
6
evaluation. The first column in Table 2 lists
the desired tool capabilities to support SoSESE comparisons, then identifies the
priority/relevance of that capability using a
“high, medium, low” rating. Next, for each
tool, an indicator shows how well that tool
provides the desired capability. The indicators
range from black (provides adequate
capability) to clear (provides very little or no
capability, or the tool requires extensive
work/programming to implement desired
capability).
Table 2. Summary of Process Model Tool Capabilities1
Vennsim
MS Project
Capability
Priority
Capability
Flowchart
Tool
Shows activities and relationships between activities
(predecessor/successor)
High
■ ■
■
Allows decomposition of activity
High
■ ■
■
Varying granularity: Allows automated roll-up/ expansion of lower level
activities
Med
■ ■
■
Shows conditional iteration
High
Shows order in which activities are performed
High
Allows identification of organization/team responsible for activities
Med
Reflects development life cycle model
Med
■
■
■
■
■
■
■
■
■
■
■
□
Flexibility: Allows easy modification of process activities/tasks for each
project/program
High
■ ■
■
Primary Orientations
Activity (describes activities used to develop product)
High
■ ■
■
Product (focus on product transformation)
Med
■ ■
□
Decision (oriented towards decision making required to develop
product)
Low
□ □
■
Context (decision based on context)
Low
□
■
■
□
■
■
■
□
Shows rework
Med
Supports extensive parallelism of activities
High
Tracks/analyzes task durations
High
1
□
■
■
■
Legend:
■
■

Provides adequate capability in indicated area
Provides some (limited) capability in indicated area (the darker the color, the more capability provided)
Provides very little or no capability in indicated area or capability requires extensive programming to implement
7
Table 2. Summary of Process Model Tool Capabilities1
Vennsim
MS Project
Capability
Priority
Capability
Flowchart
Tool
Med
□ ■
□ ■
■
■
Allows classification/grouping of activities (e.g., related to SoS
interfaces, net-centricity, development of new technologies, etc.)
Med
■ ■
□
Shows interdependencies between tasks/sets of tasks
High
Supports critical path analysis
Med
■ ■
□ ■
■
□
Shows “influences” on activity effort (what causes it to be more/less) –
e.g., management issues (multiple suppliers, distributed teams),
technology maturity, design maturity, competing business goals,
standards
High
□ ■
■
Supports comparison of multiple projects
High
Provides automated analysis of multiple projects
Med
Provides report generation capability
Med
■ ■
□ □
□ ■
■
■
■
Shows % of effort associated with activity
High
Supports Activity Based Costing (ABC)
As can be seen from Table 2, each of the
tools provides several of the high priority
capabilities desired for the SoSE analysis.
However, no single tool provided all of the
high priority features/capabilities.
The
following summarizes some of the key
features and limitations of each tool.
Flowcharts: Flowcharts provide an easyto-use way of showing key tasks/activities and
the relationships between them. It is also
possible to show organizational roles with
respect to the various tasks/activities.
However, flowcharts do not incorporate a
timeline. Therefore, it is difficult to show
events with respect to calendar time or to
show all on-going parallel tasks and the
relationships between them.
Also, flowcharts do not have a mechanism
for capturing effort or duration associated with
a specific task, again making it difficult to
view/understand the nature of parallel
activities and the percentage of overall effort
associated with these activities.
In addition, it is not easy to show activities
performed by multiple organizations (e.g.,
coordinating
requirements
changes,
architecture evolution, and problem resolution
activities across multiple stakeholders,
vendors, and suppliers) without interrupting
the flow of other activities or using multiple
connectors. Finally, there is no mechanism
within flowcharts to show “influences” on
activity effort, which is key to understanding
potential differences between SoSE and
classical SE.
MS Project: MS Project allows users to
easily enter tasks/activities in a WBS-like
format and to assign resources, durations,
start/finish dates, and costs to each activity.
However, it can be difficult to integrate longterm oversight tasks often required when
working with subcontractors or vendors and to
track resources between the oversight tasks
and other SoSE tasks. Also, it can be difficult
to show problem mitigation/resolution
activities and associated rework for resolving
8
actual problems. Lastly, as with several of the
other related business process re-engineering
tools considered for this evaluation, there is no
mechanism within MS Project to show
“influences” on activity effort, which is key to
understanding potential differences between
SoSE and classical SE.
Vensim: Vensim is a powerful system
dynamics modeling tool that allows one to
model “influences” on processes or activities
and to understand the effects as influences
increase or decrease, a key analysis capability
needed for understanding potential differences
between SoSE and classical SE. However,
Vensim does not include mechanisms to easily
identify all process steps/activities, track
activities over time, or show task predecessors
/successors.
Additional Analyses. Because no one
process modeling tool met all of the high
priority needs for the SoSE-classical SE
comparison, further process model tool
analyses were conducted. Using a Quality
Function Deployment (Blanchard et al. 1998)
type of analysis to compare the various tool
features to the desired tool capabilities, it was
found that the best solution would be the
combination of MS Project and a system
dynamics modeling tool such as Vensim. MS
Project supports the definition of activities at
varying levels of granularity and the
relationships
between
them,
shows
organizational responsibilities, reflects the life
cycle development model, handles parallelism
of tasks well, and tracks durations and costs.
The system dynamics modeling tool can then
be used to better understand the influences on
key activities in SoSE and classical SE.
Conclusions
The goal of this research was to identify
one or more process modeling tools that could
be used to support comparisons of SoSE and
classical SE. In order to better understand
each type of engineering, efforts are underway
to evaluate processes from multiple programs,
both SoSE and classical SE. Criteria for
process modeling tools to support this effort
were extracted from SoSE conferences and
workshops that identified potential areas
where SoSE and classical SE differed. Then a
survey of process modeling tools was
conducted and candidates from each type of
process modeling tool selected for further
evaluation.
Tool comparisons against the desired
process model capabilities showed that the
combined use of MS Project and a system
dynamics modeling tool such as Vensim can
capture and help analyze much of the SoSE
and classical SE information needed to
perform the desired SoSE-SE comparison.
References
Blanchard, B. S. and Fabrycky, W. J., Systems
Engineering and Analysis, Prentice Hall,
1998.
Breyfogle, F., Implementing Six Sigma
Smarter Solution Using Statistical
Methods, Second Edition, John Wiley and
Sons, 2003.
Carlock, P., and Lane, J., “System of Systems
Enterprise Systems Engineering, the
Enterprise Architecture Management
Framework, and System of Systems Cost
Estimation”, 21st International Forum on
COCOMO and Systems/Software Cost
Modeling, 2006.
Cockburn, A., Writing Effective Use Cases,
Addison-Wesley, 2001.
Department of Defense (DoD), System of
Systems
Engineering
Guide:
Considerations for Systems Engineering in
a System of Systems Environment, draft
version 0.9, 2006.
Eclipse
Process
Framework
Project,
http://www.eclipse.org/epf/, accessed on
11/16/2006.
Intelligent Systems Technology Incorporated,
Process Edge™ Enterprise Suite: User’s
Guide, Version 3.5, 2004.
9
Lane, J., "COSOSIMO October 2005
Workshop", Proceedings of the 20th Forum
on COCOMO and Software Cost
Modeling, USC CSE, 2005.
Lane, J., “COSOSIMO Tutorial”, 21st
International Forum on COCOMO and
Systems/Software Cost Modeling, 2006.
Lane, J. and Valerdi, R., “Synthesizing SoS
Concepts for Use in Cost Estimation”,
Proceedings of IEEE SMC Conference,
October 2005.
Maier, M., “Architecting Principles for
Systems-of-Systems”,
Systems
Engineering, 1:4, pp 267-284, 1998.
Microsoft, Project Standard 2003 Overview,
accessed at http://www.microsoft.com/
office/project/prodinfo/standard/overview.
mspx on 11/1/2006.
Northrop, L., et al., Ultra-Large-Scale
Systems: The Software Challenge of the
Future, Software Engineering Institute,
2006.
Object Management Group (OMG), Unified
Modeling Language, http://www.uml.org/
accessed on 1/2/2007.
Proceedings of the 2nd Annual System of
Systems
Engineering
Conference,
Sponsored by System of Systems
Engineering Center of Excellence
(SoSECE), http://www.sosece.org/, 2006.
United States Air Force (USAF) Scientific
Advisory Board (SAB), Report on Systemof-Systems Engineering for Air Force
Capability Development; Public Release
SAB-TR-05-04, 2005.
Ventana Systems Incorporated, Vensim®
Version 5 User’s Guide, 2006.
Wang, G., Valerdi, R., Lane, J. Boehm, B.,
“Towards a Work Breakdown Structure
for Net Centric System of Systems
Engineering and Management”, INCOSE
Symposium, 2006.
engineering and research activities. In this
capacity, she is currently working on a cost
model to estimate the effort associated with
system-of-system architecture definition and
integration. Prior to her research work, Ms.
Lane was a key technical member of Science
Applications
International Corporation’s
(SAIC) Software and Systems Integration
Group, She has 30 years of software system
architecting and engineering experience on
both information management and real-time
military systems.
F. Stan Settles, Ph.D., is the IBM Chair in
Engineering Management, the Director of the
Systems Architecture and Engineering
Program, the co-director of the Center for
Systems and Software Engineering, the
Director of the Engineering Management
Program, and the former Chair of the Daniel J.
Epstein Department of Industrial and Systems
Engineering at the University of Southern
California. His research and teaching interests
are in the areas of quality management,
engineering project management, and
manufacturing systems engineering.
Dr.
Settles spent 30 years at AlliedSignal
Aerospace prior to joining USC.
Barry Boehm, Ph.D., is the TRW
professor of software engineering and codirector of the Center for Systems and
Software Engineering at the University of
Southern California. He was previously in
software engineering, systems engineering,
and management positions at General
Dynamics, Rand Corp., TRW, and the
Defense Advanced Research Projects Agency,
where he managed the acquisition of more
than $1 billion worth of advanced information
technology systems. Dr. Boehm originated the
spiral model, the Constructive Cost Model,
and the stakeholder win-win approach to
software management and requirements
negotiation.
Biographies
Jo Ann Lane is currently a USC CSSE
Principal supporting software and systems
10