When Do You Need Systems of
Systems Engineering:
A Quantitative Analysis
Jo Ann Lane
17 March 2009
University of Southern California
Center for Systems and Software Engineering
Overview
•
•
•
•
•
•
Key definitions
Scope of research
Methodology
Model implementation
Results of research
Conclusions and future work
March 2009
USC CSSE Annual Research Review
2
What is a “System of Systems”?
• Very large systems developed by creating a framework
or architecture to integrate constituent systems
• SoS constituent systems independently developed and
managed
–
–
–
–
New or existing systems in various stages of development/evolution
May include a significant number of COTS products
Have their own purpose
Can dynamically come and go from SoS
• SoS exhibits emergent behavior not otherwise
achievable by component systems
• Typical domains
– Business: Enterprise-wide and cross-enterprise integration to support
core business enterprise operations across functional and geographical
areas
– Military: Dynamic communications infrastructure to support operations
in a constantly changing, sometimes adversarial, environment
Based on Mark Maier’s SoS definition [Maier, 1998]
March 2009
USC CSSE Annual Research Review
3
Types of SoS
•
Virtual [Maier, 1998]
– Lacks a central management authority and a clear SoS purpose
– Often ad hoc and may use a service-oriented architecture where
the constituent systems are not necessarily known
•
Collaborative [Maier, 1998]
– Constituent system engineering teams work together more or
less voluntarily to fulfill agreed upon central purposes
– No SoSE team to guide or manage activities of constituent
systems
•
Acknowledged [Dahmann, 2008]
– Have recognized objectives, a designated manager, and
resources at the SoS level (SoSE team)
– Constituent systems maintain their independent ownership,
objectives, funding, and development approaches
•
This research
focused on
identifying the
“home-ground”
for these two
types of SoSs...
Directed [Maier, 2008]
– SoS centrally managed by a government, corporate, or Lead
System Integrator (LSI) and built to fulfill specific purposes
– Constituent systems maintain ability to operate independently,
but evolution subordinated to centrally managed purpose
March 2009
USC CSSE Annual Research Review
4
Scope of Research
• Research question
– When is it cost effective to establish and use a system of
systems engineering (SoSE) team to oversee and guide the
evolution of a system of systems (SoS)?
• Hypothesis
– There exists a threshold where it is more cost effective to
manage and engineer capability changes to an SoS using an
SoSE team and this threshold can be determined by modeling
the SoS system complexity and desired capability
interdependency characteristics.
Focus is on software-intensive SoSs owned by the
United States Department of Defense (DoD)...
March 2009
USC CSSE Annual Research Review
5
Statement of Topic and Contribution
(continued)
• Research contribution
– Provides guidance to DoD leadership with respect to the
management of sets of inter-related systems that are functioning
as a system of systems.
– Guidance also applies to SoSs in other domains that are
managed as collaborative or acknowledged SoSs
– Model for management and engineering guidance also provides
• A method for conducting trade-off analyses for different approaches
for implementing a given SoS capability for a given SoS
• A model that can evolve into an SoSE cost model through
calibration for a given SoS or SoS domain
• A cost model that can better model complex systems
March 2009
USC CSSE Annual Research Review
6
Methodology
• Using COSYSMO, developed a process model that can
compare the SoS management strategies as SoS
characteristics are varied
– SoS size (number of constituent systems)
– Size of SoS capability (number of equivalent nominal requirements)
– Scope of SoS capability (number of constituent systems affected by
SoS capability)
– Constituent system volatility (level of constituent system change being
engineered at the same time as SoS capability)
• Process model based on data from
– 18 large-scale DoD SoS programs
– 16 DoD systems that participate as constituent systems in one or more
SoSs
• Analyze model outputs to determine under what conditions
an SoSE team is cost effective
March 2009
USC CSSE Annual Research Review
7
SoSE Process Model Overview
•
Purpose
–
•
Estimate and compare the effort required to implement an SoS capability using
two different management approaches
•
Collaborative (no SoSE team)
•
Acknowledged (SoSE with limited authority/control)
Assumptions and constraints
–
–
–
–
–
–
–
March 2009
All constituent systems currently exist and have their own evolutionary paths
based on system-level stakeholder needs/desires
Model assumes SoSE and traditional SE teams are using relatively mature
processes
SoS capabilities are software-intensive
No SoS capability/requirements volatility
SoS internal volatility represented by constituent system volatility
No accommodation of schedule factors or the asynchronous nature of SoS
constituent system upgrades
Management of SoS internal interfaces reduces complexity for systems
USC CSSE Annual Research Review
8
Systems Engineering Requirements Categories
•
Requirements related to SoS capabilities
a) Acknowledged SoS: Initially engineered at SoS level by SoSE
team with support from constituent system engineers for those
systems impacted by the SoS capability, then allocated to
constituent systems for further SE
b) Collaborative SoS: Not engineered at the SoS level, but must be
engineered fully at the constituent system level through
collaborative efforts with other constituent system engineers
•
Non-SoS requirements related to constituent system
stakeholder needs
– Must be monitored by SoSE team to identify changes that might
adversely impact SoS
– Represents on-going volatility at the constituent system level that
is occurring in parallel with SoS capability changes
March 2009
USC CSSE Annual Research Review
9
SoSE Model Structure
Focus is on softwareintensive SoSs
owned by the US
DoD, the number and
volatility of
constituent systems
within an SoS, and
the complexity of
typical capability
enhancements to the
SoS...
March 2009
USC CSSE Annual Research Review
10
Overview of SoSE SDM Flow
Conversion to
COSYSMO size units
System
Capability
March 2009
Calculations based on SoS
characteristics/size and capability
implementation approach using
COSYSMO algorithm
Equivalent
set of
“sea-level”
requirements
USC CSSE Annual Research Review
Effort using an
“acknowledged”
SoSE team
Effort for a
“collaborative”
SoS
11
Model Parameters by SDM Construct
• Stocks
• Converter Parameters
– Inputs
• SoS Equivalent Requirements
– Outputs
• SoSE Effort
• SoS Upgrade Effort with
SoSE
• SoS Upgrade Effort without
SoSE
• Flows
–
–
–
–
Capability Rate
SoSE Effort Rate
SE Effort Rate with SoSE
SE Effort Rate without SoSE
– COSYSMO effort multipliers
• COSYSMO SoSE EM
• COSYSMO SE EM with
SoSE
• COSYSMO SE EM without
SoSE
• COSYSMO SE EM
– SoS complexity factors
• Number of systems in SoS
• Number of systems affected
by capability
• Average system rate of
change
General Form of COSYSMO Equation
Effort (person months) = [38.55 * EM * (size)1.06] / 152
March 2009
USC CSSE Annual Research Review
12
SoSE Effort Multiplier
2.50
March 2009
USC CSSE Annual Research Review
13
Effort Multiplier for SoSE Monitoring of
Constituent System Requirements
0.47
March 2009
USC CSSE Annual Research Review
14
SE Effort Multiplier for SoS Requirements
with SoSE Support
1.06
March 2009
USC CSSE Annual Research Review
15
SE Effort Multiplier SoS Requirements
without SoSE Support
1.79
March 2009
USC CSSE Annual Research Review
16
SE Effort Multiplier for System-Specific
(Non-SoS) Requirements
0.72
March 2009
USC CSSE Annual Research Review
17
Effort Calculations
SoSE Effort
SoSE Effort = 38.55*[((SoSCR/SoSTreq)*(SoSTreq)1.06 *EMSoS-CR)+
((SoSMR/SoSTreq)*(SoSTreq)1.06 * EMSoS-MR)/152]
Where:
Total SoSE requirements = SoS Capability Requirements + SoS “Monitored” Requirements
SoS “monitored” reqs = [∑SE non-SoS requirements being addressed current upgrade cycles
for all SoS constituent systems] * “Oversight Factor”
“Oversight Factor” = 5% , 10%, 15% (these values are based on expert judgment from
various CSSE affiliates and the SoS SE Guidebook team)
Based on COCOMO II approach for combining components with different
EMs (SoS changes and Constituent System oversight)
March 2009
USC CSSE Annual Research Review
18
Effort Calculations (continued)
Single System Effort with Support from SoSE Team
Total single system reqsw-SoSE = SoS requirements allocated to system + SE reqs in
upgrade cycle
Single system SE Effort with SoSE Team
= 38.55*[1.15*( (SoSCSalloc / CSTreqSoSE)*( CSTreqSoSE)1.06* EMCS-CRwSOSE) +
(CSnonSoS / CSTreqSoSE)*( CSTreqSoSE)1.06* EMCSnonSOS] /152
Based on COCOMO II approach for combining components with different EMs plus
including a 15% “tax” to support SoSE team in their engineering effort for the SoSE
requirements. 15% represents half of the system design effort in the EIA 632 tasks.
March 2009
USC CSSE Annual Research Review
19
Effort Calculations (continued)
Single System Effort with No SoSE Team Support
Total single system reqs wo-SoSE = SoSE capability reqs + SE non-SoS requirements
Single system SE Effort without SoSE Team =
38.55*[(( SoSCR / CSTreqwoSoSE)*( CSTreqwoSoSE)1.06* EMCS-CRnSOSE) +
((CSnonSoS / CSTreqwoSoSE)*( CSTreqwoSoSE)1.06* EMCSnonSOS)] /152
Based on COCOMO II approach for combining components
with different EMs (SoS changes and non-SoS changes)
March 2009
USC CSSE Annual Research Review
20
Range of SoS Complexity Factor Values
SoSE Model
Parameter
Description
Range of Values
SoS Size
Number of constituent systems within
the SoS
2-200
SoS Capability Size
Number of equivalent nominal
requirements as defined by COSYSMO
1-1000
Constituent System
Volatility
Number of non-SoS changes being
implemented in each constituent
system in parallel with SoS capability
changes
0-2000
Scope of SoS
Capability
Number of constituent systems that
must be changed to support capability
One to SoS Size (total
number of constituents
systems within the SoS)
SoSE Oversight
Factor
Oversight adjustment factor to capture
SoSE effort associated with monitoring
constituent system non-SoS changes
5%, 10%, and 15%
March 2009
USC CSSE Annual Research Review
21
Results of Research
Scenario 2 (SoS Size Varies)
Scenario 1 (SoS Size Varies)
Relative Cost of Collaborative and Acknowledged SoSE
Capability Affects Half of the Systems
System Volatility = 100 Reqs and SoS Capability = 100 Reqs
Relative Cost of Collaborative and Acknowledged SoSE
Capability Affects Half of the Systems
System Volatility = 100 Reqs and SoS Capability = 50 Reqs
1800.00
800.00
Savings (Person Months)
Savings (Person Months)
1500.00
1200.00
OSF 5%
900.00
OSF 10%
600.00
OSF 15%
300.00
0.00
0
50
100
150
200
250
600.00
OSF 5%
400.00
OSF 10%
200.00
OSF 15%
0.00
0
50
Scenario 3 (SoS Size Varies)
200
250
Scenario 4 (SoS Size Varies)
Relative Cost of Collaborative and Acknowledged SoSE
Capability Affects Half of the Systems
System Volatility = 100 Reqs and SoS Capability = 25 Reqs
Relative Cost of Collaborative and Acknowledged SoSE
Capability Affects One-Fourth of the Systems
System Volatility = 100 Reqs and SoS Capability = 100 Reqs
400.00
2000.00
300.00
200.00
OSF 5%
100.00
OSF 10%
0.00
OSF 15%
0
50
100
150
200
250
Savings (Person
Months)
Savings (Person Months)
150
Number of Systems
Num ber of System s
-100.00
100
-200.00
-300.00
1600.00
1200.00
OSF 5%
800.00
OSF 10%
400.00
OSF 15%
0.00
-400.00 0
-200.00
100
150
200
250
Number of Systems
Number of Systems
March 2009
50
USC CSSE Annual Research Review
22
Results of Research (continued)
Scenario 6 (SoS Size Varies)
Scenario 5 (SoS Size Varies)
Relative Cost of Collaborative and Acknowledged SoSE
Capability Affects All of the Systems
System Volatility = 2000 Reqs and SoS Capability = 100 Reqs
Relative Cost of Collaborative and Acknowledged SoSE
Capability Affects Half of the Systems
System Volatility = 2000 Reqs and SoS Capability = 100 Reqs
2000.00
0
50
100
150
200
250
-5000.00
OSF 5%
OSF 10%
OSF 15%
-10000.00
Savings (Person Months)
Savings (Person Months)
0.00
-15000.00
0.00
-2000.00
0
50
100
150
200
250
OSF 5%
-4000.00
OSF 10%
OSF 15%
-6000.00
-8000.00
-10000.00
Num ber of System s
Num ber of System s
Scenario 7-a (SoS Size = 10)
Scenario 7-b (SoS Size = 100)
Relative Cost of Collaborative and Acknowledged SoSE
SoS Capability Scope Varies
System Volatility = 1000 Reqs and SoS Capability = 1000 Reqs
Relative Cost of Collaborative and Acknowledged SoSE
SoS Capability Scope Varies
System Volatility = 1000 Reqs and SoS Capability = 1000 Reqs
25000.00
1000.00
500.00
OSF 5%
0.00
OSF 10%
0
1
2
3
4
5
6
7
8
9
10
-500.00
-1000.00
11
12
OSF 15%
Savings (Person
Months)
Savings (Person Months)
1500.00
20000.00
15000.00
OSF 5%
10000.00
OSF 10%
5000.00
OSF 15%
0.00
-5000.00 0
-1500.00
20
40
60
80
100
120
Number of Systems Affected by Capability
Number of Systems Affected by Capability
March 2009
USC CSSE Annual Research Review
23
Results of Research (continued)
Scenario 8-b (SoS Size = 100)
Scenario 8-a (SoS Size = 10)
Relative Cost of Collaborative and Acknowledged SoSE
SoS Capability Scope Varies
System Volatility = None and SoS Capability = 1000 Reqs
Relative Cost of Collaborative and Acknowledged SoSE
SoS Capability Scopre Varies
System Volatility = None and SoS Capability = 1000 Reqs
1000.00
OSF 5%
500.00
OSF 10%
0.00
OSF 15%
0
1
2
3
4
5
6
7
8
9
10
11
12
-500.00
Savings (Person
Months)
Savings (Person Months)
1500.00
25000.00
20000.00
OSF 5%
15000.00
OSF 10%
10000.00
OSF 15%
5000.00
0.00
0
20
40
60
80
100
120
-1000.00
Number of SYstems Affected by Capability
Num ber of System s Affected by Capability
Scenario 9 (SoS Size = 10)
Scenario 10 (SoS Size = 5)
Relative Cost of Collaborative and Acknowledged SoSE
SoS Capability Scope Varies
System Volatility = 1000 and SoS Capability = 1 Req
Relative Cost of Collaborative and Acknowledged SoSE
SoS Size = 5 SoS Capability Scope Varies
System Volatility = 1000 Reqs and SoS Capability = 1000 Reqs
100.00
0.00
0
1
2
3
4
5
6
7
8
9
10
11
12
OSF 5%
OSF 10%
-100.00
OSF 15%
-200.00
Savings (Person
Months)
Savings (Person Months)
500.00
0.00
OSF 5%
0
2
3
4
5
6
OSF 10%
OSF 15%
-500.00
-1000.00
-300.00
Num ber of System s Affected by Capability
Num ber of System s Affected by Capability
March 2009
1
USC CSSE Annual Research Review
24
Results of Research (continued)
Scenario 12 (SoS Size = 5)
Scenario 11 (SoS Size = 5)
Relative Cost of Collaborative and Acknowledged SoSE
SoS Size = 5 SoS Capability Scope Varies
System Volatility = 1000 and SoS Capability = 1 Req
Savings (Person
Months)
200.00
0.00
-200.00 0
1
2
3
4
5
6
OSF 5%
OSF 10%
-400.00
OSF 15%
-600.00
-800.00
Savings (Person Months)
Relative Cost of Collaborative and Acknowledged SoSE
SoS Size = 5 SoS Capability Scope Varies
System Volatility = None and SoS Capability = 1000 Reqs
0.00
0
2
3
4
5
6
OSF 5%
-80.00
OSF 10%
OSF 15%
-120.00
-160.00
Number of Systems Affected by Capability
March 2009
1
-40.00
Number of Systems Affected by Capability
USC CSSE Annual Research Review
25
Conclusions
When is it cost effective to
create and empower an SoSE
team to oversee and guide the
evolution of an SoS?
There exists a threshold where it is more
cost effective to manage and engineer
changes to an SoS using an SoSE team and
this threshold can be determined by
modeling the SoS’ interdependency and
complexity characteristics.
SoSE Model
March 2009
Model parameters:
SoS size
Scope/size of SoS change
CS volatility
SoSE oversight
USC CSSE Annual Research Review
26
Conclusions (continued)
• SoSE team is cost effective when
– SoS contains more than a “few” systems
– SoS capability changes typically affect a “significant
percentage” of constituent systems
– SoS capability requirements are a “significant percentage”
of the total requirements being addressed by constituent
systems in an upgrade cycle
– SoS oversight activities and the rate of capability
modifications/changes being implemented are sufficient to
keep an SoSE team engaged (i.e., little-to-no slack time)
• SoSE team is NOT cost effective when
– The number of systems in an SoS is “small”
– The constituent system volatility is high and the SoS
changes are small
March 2009
USC CSSE Annual Research Review
27
Conclusions (continued)
• The “oversight factor” (the amount of effort spent by
the SoSE team to monitor non-SoS changes in the
constituent systems) is a key factor in determining
the cost effectiveness of the SoSE team
– More work is needed to determine a more accurate
“oversight factor”
– This factor may be variable across multiple SoSs
• There may be reasons other than cost to engage an
SoSE team
– Importance of SoS
– Critical SoS performance requirements requiring extensive
analysis at the SoS level
March 2009
USC CSSE Annual Research Review
28
Future Work
• Expand SoSE model to
– Include schedule factors to allow trade-offs between
“faster” and “cheaper”
– Include quality factors based on complexities and the
resulting rework due to inadequate SoS engineering
– Allow users to specify specific constituent system
configurations to allow capability alternative trade-offs
• Investigate the factors in going from an
Acknowledged SoS to a Directed SoS
March 2009
USC CSSE Annual Research Review
29
Backup Charts
March 2009
USC CSSE Annual Research Review
30
Traditional SE and SoSE Activities
Translating
Translating
Translating
capability
capability
capability
objectives
objectives
objectives
Understanding
Understanding
systems
Understanding
systems&&
relationships
systems &
relationships
(includes
plans)
relationships
(includes
plans)
Orchestrating
Orchestrating
Orchestrating
upgrades
upgrades
upgrades
to
to
SoS
toSoS
SoS
Addressing
Addressing
new
Addressing
new
requirements
requirements
requirements
&
solution
&
options
& options
options
Assessing
Assessing
(actual)
Assessing
(actual)
performance
performance
performance
tototo
capability
capability
capability
objectives
objectives
objectives
Developing,
Developing,
Developing
evolving
and
evolving
and
& evolving
maintaining
maintaining
SoS
SoS
design/arch
SoSarchitecture
design/arch
Monitoring
Monitoring
Monitoring
&&assessing
assessing
&
assessing
changes
changes
changes
External Environment
Traditional SE
(Defense Acquisition Guide
[DoD, 2006] View)
March 2009
SoSE
(SoS SE Guidebook View Based on
Interviews and Analysis of
18 DoD SoSs in Various Stages)
USC CSSE Annual Research Review
31
Key COSOSIMO Research Findings
• Limitations of COSYSMO for “Directed” SoSE effort
estimation
– Missing cost factors
•
•
•
•
•
Cost/schedule compatibility of proposed SE approach
Level of overall risk resolution
Number of constituent systems and associated organizations
Constituent system maturity and stability
Constituent system readiness
– Need to adjust for SoSE oversight of constituent system SE
– Need ability to assign different EMs to various parts of SE
March 2009
USC CSSE Annual Research Review
32
Using System Dynamics Models to Explore
Alternatives or Influences in the Development of Large
Software-Intensive Systems
• System dynamics modeling
tool: visual modeling tools that
allow one to conceptualize,
simulate and analyze models
of dynamic systems and
processes
• Consist of causal loops or
stock and flow diagrams
• Models are executable,
allowing use to explore
behaviors of the model as
variables representing process
influences are changed
March 2009
• Examples:
– Hybrid/plan-driven ICM [Madachy
et al., 2007]
– Intergovernmental collaboration
[Cresswell et al., 2002]
– Inter-Organizational Baseline
Alignment [Greer et al., 2005]
– Requirements volatility [Ferreira,
2002]
– Under-allocation of resources in
early phases of a project [Black
and Repenning, 2001]
– Interactions between concurrently
developed projects [Ford and
Sterman, 2003]
USC CSSE Annual Research Review
33
Model Validity Rationale
• SoSE model description
– Comparison model based on a modified version of the validated
academic systems engineering cost model, COSYSMO
– Modifications based upon key findings of the OSD SoSE case studies
• Validation goal: Show that the SoSE cost model is a valid method
conducting sensitivity analyses for two different SoS management
strategies
– Collaborative
– Acknowledged
• Not part of the validation goal: The estimation of actual effort
associated with a specific SoS or a given set of SoSs
– The calibration/validation of the SoSE model for this purpose is left for
future work
March 2009
USC CSSE Annual Research Review
34
Model Validity Rationale (continued)
• Validity argument
– COCOMO II and Academic COSYSMO are multiple regression models
that have been calibrated and validated with actual data from primarily
DoD programs
– Academic COSYSMO calibration data contains 3 SoS data points
– Most other COSYSMO calibration data points interface to other systems,
which implies that they are part of one or more SoSs
– SoSE model was developed using
• Academic COSYSMO that includes ability to distribute effort across SE phases
• Locally validated COSYSMO extension to adjust effort for reuse/oversight of
evolving system components
• COCOMO II technique for using multiple effort multipliers to characterize
components with different characteristics and complexities
– SoSE model parameters
• Based on ranges of size drivers determined through case studies and surveys
• Uses nominal cost driver values unless reasons identified in SoSE or SE
survey data to indicate otherwise
• Resulting in a relative comparison of the two management approaches
March 2009
USC CSSE Annual Research Review
35
Model Validity Rationale (continued)
• Validity argument (continued)
– The prediction accuracies (PRED factors) are
• COCOMO [Clark and Reifer, 2007]
– PRED(30) = 75% (with no stratification of projects)
– PRED(30) = 80% (with stratification of projects)
• COSYSMO [Valerdi, 2005]
– PRED(30) = 75% (with stratification of projects)
– PRED(30) = 85% (anecdotal evidence from local calibrations)
– The OSD SoSE cases studies show that SoS systems engineers perform
the same types of activities as addressed by the SE cost model,
COSYSMO
– The OSD SoSE case studies identify differences between SoSE and SE
for a single system and most of these differences are with respect to
parameters in the SE cost model, COSYSMO
– There exists a local (single organization) calibrated and validated method
within COSYSMO to estimate effort for oversight of related/interfacing
systems or reusable components [Wang et al, 2008]
March 2009
USC CSSE Annual Research Review
36
References
Ackoff, R. (1971); “Towards a System of Systems Concepts”, Management Science, Vol 17, No. 11, Theory Series, pp. 661-671.
ANSI/EIA (1999). ANSI/EIA-632-1988 Processes for Engineering a System.
Berry, B. (1964); “Cities as Systems within Systems of Cities”, The Regional Science Association Papers, Volume 13..
Black, L. and Repenning, N. (2001); “Why Firefighting is Never Enough: Preserving High-Quality Product Development”,
System Dynamics Review, Vol. 17, No: 1, pp. 33-62
Blanchard, B. and Fabrycky, W. (1998). Systems Engineering and Analysis, Prentice Hall.
Boehm, B., Abts, C., Brown, A. W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D. J. and Steece, B. (2000).
Software Cost Estimation With COCOMO II, Prentice Hall.
Boehm, B. and Lane J. (2006) "21st Century Processes for Acquiring 21st Century Software-Intensive Systems of Systems."
CrossTalk - The Journal of Defense Software Engineering, Vol. 19, No. 5, pp.4-9.
Boehm, B., Valerdi, R., Lane, J., Brown, A., (2005) “COCOMO Suite Methodology and Evolution,” CrossTalk - The Journal of
Defense Software Engineering, Vol. 18, No. 4, pp. 20-25, April 2005.
Cocks, D. (2006); “How Should We Use the Term “System of Systems” and Why Should We Care?”, Proceedings of the 16th
Annual INCOSE International Symposium.
Cresswell, A. et al. (2002); "Modeling Intergovernmental Collaboration: A System Dynamics Approach", Proceedings of the
35th Annual Hawaii International Conference on System Sciences.
Dahmann, J. and Baldwin. K. (2008); “Understanding the Current State of US Defense Systems of Systems and the
Implications for Systems Engineering”, Montreal, Canada: Proceedings of the IEEE Systems Conference, 7-10 April.
Department of Defense (DoD) (2006); Defense Acquisition Guidebook, Version 1.6, accessed at http://akss.dau.mil/dag/ on
2/2/2007.
Department of Defense (DoD) (2008); Systems Engineering Guide for System of Systems, version 1.0.
Dorner, D. (1996); The Logic of Failure, Metropolitan Books.
Ferreira S. (2002); Measuring the Effects of Requirements Volatility on Software Development Projects. Ph.D. Dissertation,
Arizona State University.
Finley, J. (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference
Ford D. and Sterman J. (2003); "Iteration Management for Reduced Cycle Time in Concurrent Development Projects",
Concurrent Engineering Research and Application (CERA) Journal.
Friedman, T. (2005), The World is Flat: A Brief History of the Twenty-First Century, Farrar, Straus and Giroux, New York.
Greer, D., Black, L., Adams, R. (2005), "Improving Inter-Organizational Baseline Alignment in Large Space System
Development Programs", Proceedings of IEEE Aerospace Conference.
Highsmith, J. (2000); Adaptive Software Development: A Collaborative Approach to Managing Complex Systems, Dorset
House Publishing.
March 2009
USC CSSE Annual Research Review
37
References (continued)
INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP-2003-002-03.
isee Systems (2007), "iThink", http://www.iseesystems.com/Softwares/Business/ithinkSoftware.aspx accessed on
2/10/2007.
ISO/IEC (2002). ISO/IEC 15288:2002(E) Systems Engineering - System Life Cycle Processes.
Ferreira S. (2002); Measuring the Effects of Requirements Volatility on Software Development Projects. Ph.D. Dissertation,
Arizona State University.
Finley, J. (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference
Ford D. and Sterman J. (2003); "Iteration Management for Reduced Cycle Time in Concurrent Development Projects",
Concurrent Engineering Research and Application (CERA) Journal.
Friedman, T. (2005), The World is Flat: A Brief History of the Twenty-First Century, Farrar, Straus and Giroux, New York.
Greer, D., Black, L., Adams, R. (2005), "Improving Inter-Organizational Baseline Alignment in Large Space System
Development Programs", Proceedings of IEEE Aerospace Conference.
Highsmith, J. (2000); Adaptive Software Development: A Collaborative Approach to Managing Complex Systems, Dorset
House Publishing.
INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP-2003-002-03.
isee Systems (2007), "iThink", http://www.iseesystems.com/Softwares/Business/ithinkSoftware.aspx accessed on
2/10/2007.
ISO/IEC (2002). ISO/IEC 15288:2002(E) Systems Engineering - System Life Cycle Processes.
Kreitman, K.(1996), "From 'The Magic Gig' to Reliable Organizations: A New Paradigm for the Control of Complex
Systems", Symposium on Complex Systems Engineering, http://cs.calstatela.edu/wiki/index.php/
Symposium_on_Complex_Systems_Engineering, accessed on 1/11/2007.
Krygiel, A. (1999); Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33
Lane, J. and Boehm, B. (2007); Modern Tools to Support DoD Software Intensive System of Systems Cost Estimation: A
DACS State-of-the-Art Report, Data and Analysis Center for Software, DACS Report Number 347336.
Lane, J. and Valerdi, R., (2007); "Synthesizing System-of-Systems Concepts for Use in Cost Estimation", Systems
Engineering, Vol. 10, No. 4.
Lu, S. (2003); Engineering as Collaborative Negotiation: A New Paradigm for Collaborative Engineering,
http://wisdom.usc.edu/ecn/about_ECN_what_is_ECN.htm accessed on 2/14/2007.
Madachy, R., B. Boehm, and J. Lane (2007); “Assessing Hybrid Incremental Processes for SISOS Development”, Software
Process: Improvement and Practice, Vol. 12, Issue 5, pp. 461-473.
Maier, M. (1998); “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267-284)
March 2009
USC CSSE Annual Research Review
38
References (continued)
NAVSTAR Global Positioning System Joint Program Office, http://gps.losangeles.af.mil/ , accessed on 12/6/2006.
Northrop, L., et al. (2006); Ultra-Large-Scale Systems: The Software Challenge of the Future, Software Engineering Institute.
Pinney, B. (2001); Projects, Management, and Protean Times: Engineering Enterprise in the United States, 1870-1960, PhD
Dissertation, Massachusetts Institute of Technology.
Pressman, J. and Wildavsky, A. (1973); Implementation: How Great Expectations in Washington are Dashed in Oakland; Or, Why
It’s Amazing that Federal Programs Work at All, This Being a Saga of the Economic Development Administration as Told by
Two Sympathetic Observers Who Seek to Build Morals on a Foundation of Ruined Hopes, University of California Press.
Rechtin, E. (1991); Systems Architecting: Creating & Building Complex Systems, Prentice Hall.
SEI (2001), Capability Maturity Model Integration (CMMI), CMU/SEI-2002-TR-001.
Sheard, S. (2006), "Foundations of Complexity Theory for Systems Engineering of Systems of Systems", Proceedings of the IEEE
Conference on System of Systems Engineering.
United States Air Force (USAF) Scientific Advisory Board (SAB) (2005); Report on System-of-Systems Engineering for Air Force
Capability Development; Public Release SAB-TR-05-04
Valerdi, R. (2005); Constructive Systems Engineering Cost Model. PhD. Dissertation, University of Southern California.
Valerdi, R. and Wheaton, M. (2005); "ANSI/EIA 632 as a Standardized WBS for COSYSMO", AIAA-2005-7373, Proceedings of
the AIAA 5th Aviation, Technology, Integration, and Operations Conference, Arlington, Virginia.
Wang, G., Valerdi, R., Ankrum, A., Millar, C., and Roedler, G. (2008), "COSYSMO Reuse Extension", Proceedings of the 18th
Annual International Symposium of INCOSE, The Netherlands.
March 2009
USC CSSE Annual Research Review
39