25th International Forum on COCOMO
and
Systems Software Cost Modeling
Systems Engineering Desk Reference
An Aid for Cost Estimators
2-5 November 2010
University of Southern California
Los Angeles CA
Sherry Stukes
Henry Apgar
Jet Propulsion Laboratory/
California Institute of Technology
Integrated Ground Data Systems
Ground Software Systems Engineering
4800 Oak Grove Drive MS 301-225
Pasadena CA
818.393.7517 (o)
805.402.8664 (c)
sherry.a.stukes@jpl.nasa.gov
President
MCR Technologies LLC
390 N Sepulveda Blvd
Suite 1050
El Segundo CA
424.218.1616 (o)
805.402.4232 (c)
hapgar@mcri.com
Copyright 2010. All rights reserved.
Background
 SSCAG* Systems Subgroup Desk Reference product
 Represents industry “Best Practices”
 Useful to Cost Estimators
 Organized into six sections
 Contributors
 NASA (JPL, LaRC, MSFC)
 SAIC
 MCR Technologies
 USAF (SMC, AFCAA)
 USC
 Raytheon
 Design for Value
 Model Vendors (Galorath, PRICE Systems)
*SSCAG (Space Systems Cost Analysis Group) is an International working group comprised of member organizations that
develop estimating products for the space industry. SSCAG currently has four active subgroups: Hardware, Software, Risk,
and Systems, supported by members from industry, Government, and the academic community.
2
Desk Reference Overview
 System Engineering Desk Reference Content
 Document Overview
 How to Predict and Evaluate Systems Engineering and SoS Costs
 Definitions
 Rules of Thumb
 “Top 10”Tools
 Lessons Learned and Estimating Examples
 Sources
 RedStar Library
 Constellation Program
 Model vendor research
 University research
 Contractor organizations
3
Defining our Terms
 System of Systems (SoS) is a collection of
task-oriented or dedicated systems that pool
their resources and capabilities together to
obtain a new, more complex, 'meta-system'
which offers more functionality and
performance than simply the sum of the
constituent systems.”
SOS
Systems
Subsystems
Assemblies
 System Engineering can be considered to include the pure
engineering efforts to ensure that a number of subsidiary
elements function together properly but also project/program
management, integration and test, missions assurance and
ground support elements sometimes called “SEPM” or Systems
Engineering and Project/Program. The definition can be
adapted to lower levels, including subsystems and assemblies.
4
SoS Hierarchy
 The hierarchal aspect of SoS is
reflected in the fact that
depending on how one defines
system, almost any integration
activity can be tagged as SoS
 An example of SoS hierarchy is
the crew launch system for the
NASA Constellation Program.
 System of Systems” consists of the Ares I
launch vehicle system and the Orion
crew capsule system
 These systems themselves are comprised
of multiple systems
* Reference – materials submitted by Andy Prince,
NASA Marshall Space Flight Center.
5
Rules of Thumb

Model Development

 Requirements are king.
More parameters increase the explanatory power of the model, but too
many parameters make the model too complex to use and difficult to
calibrate.
 Not all requirements are created equal.
 Reuse is not free.
 Operational Scenarios may come first, but requirements will
 Break the problem and analysis into phases


ultimately describe the system.
over time; the right amount of granularity is
important.
 Don't double dip.

Let available data drive the application boundaries of the model.
 Nominal is the norm.

Design the rating scale according to the phenomenon being modeled.
 If you're estimating a large project, personnel capability is Nominal.

Some system characteristics are more likely to be cost penalties than
cost savings.
 Most of your off-Nominal cost drivers should match your last
 Find your sea level.
project.
 If you're going to sin, sin consistently.
Model Calibration
 All calibrations are local.

Calibrations fix chronic errors in over- or underestimation.

Be skeptical of data that you did not collect.

For every parameter in the model, 5 data points are required for the
calibration.

Don’t do more analysis than the data is worth.

You need less data than you think, you have more data than you think.
 Use a combination of models to estimate total system cost.
 Avoid overlap between models.
 Estimate using multiple methods (analogy, parametric, etc.).

Estimation
 Estimate early and often.
 Experts all disagree forever. Bound the options they are given to
evaluate.
 People are generally optimistic.
Model Usage

A model is not reality.

All models are wrong, but some of them are useful.

Begin with the end in mind.
6
“Top 10” Tools
Version
SE Defined
LCC Phase
Levels
WBS Defined
SE Est. Method
Price ES
System Engineering Process
A-F
Subsystem,
System
User Defined
Imbedded Algorithm
Academic COSYSMO
algorithm
Price True
Planner
System Engineering
A-D
SoS
EIA/ANSI 632
WBS
SEER -H
System Engineering and
Integration
A-F
Subsystem,
System
User Defined
Imbedded Algorithm
System Engineering & Integration
A-E
System
Yes - Configurable
CER by mission type
System Engineering (w/PM)
C,D
System
Yes – Defined
CER
USCM
Systems Engineering (w/in
Program Level Cost)
A-D
System
Yes – Configurable
CER by Spacecraft
type
NICM (both
models)
Systems Engineering (estimated
but not def.)
B,C,D thru
L+30
System
Yes - Defined
CER in two forms
COCOMO
Software Engineering in the
System Context (Waterfall WBS)
Waterfall and
Mbase
System
Software
Waterfall WBS and
MbaseWBS
COCOMO algorithm
COSYSMO
for SE
Systems Engineering Effort
A-D
System,
Systems Eng
EIA/ANSI 632
WBS
academic COSYSMO
algorithm
COSYSMO
for SoSE
Systems of Systems Engineering
Effort
C-E
SoS
DoD SEGuide for
SoS
academic COSYSMO
algorithm
7
NAFCOM
SSCM07
Cost Factors
- NASA SOS Cost Modeling
 System-of-Systems Cost = 0.07411*(DDTE$)0.9993*(1.09585)M/LV
where DDTE$ = Total DDTE* Cost, in 2006$,
M/LV = Manned or Launch Vehicle (Yes = 1, No = 0)
System-of-Systems
Actual Vs. Estimated
 Good Quality Metrics
= 93.1%
 SPE = 33.2%

$3,000
Estimated (2006 $M)
R2
$3,500
$2,500
$2,000
$1,500
$1,000
$500
$0
$0
$500
$1,000 $1,500 $2,000 $2,500 $3,000 $3,500
Actual (2006 $M)
*DDTE – Design Development Test and Evaluation
8
Apollo
Atlas II
Brilliant Pebbles
HST
ISS
Peacekeeper
Pioneer Venus
Saturn IB
Saturn V
Shuttle
Skylab
Titan IV
Systems of
Systems Cost
$1,654.1
$234.6
$77.8
$113.3
$3,646.4
$265.6
$24.9
$325.1
$603.9
$597.8
$334.1
$516.5
Total DDT&E
Cost
% of DDTE
$18,449.0
8.97%
$3,040.4
7.71%
$1,100.2
7.07%
$1,592.6
7.12%
$36,027.4
10.12%
$10,006.7
2.65%
$306.3
8.13%
$3,776.7
8.61%
$10,333.9
5.84%
$18,152.6
3.29%
$4,127.7
8.09%
$5,178.0
9.97%
2
WBS Description / Notes
System of Systems Level
Systems Engineering,
Integration and Test and Top
Level Program Management
of Architecture
Space Segment
Assets in orbit or
interplanetary
2.1.3
2.1.3.2
Crew Expolration Vehicle
(CEV)
Command Module (CM)
CM PMP
2.1.4
2.1.4.1
Orion Integration, Assy., &
C/O
Orion SE&I
Orion PM
Orion STO
Orion GSE
Service Module (SM)
SM SE IT PM
SPACE
2.1
2.1.4.2
SM PMP
2.2
2.2.1
2.2.2
International Space Station
ISS Communications
TDRS Space Segment
3
3.1
3.1.1
3.1.2
Launch Segment
CLV
First Stage
Upper Stage
3.1.2.2
3.1.2.1
4
4.1
4.1.1
4.1.2
4.1.3
4.1.4
4.2
* Reference – data collected and analyzed by Dr. Christian Smart,
SAIC, under contract to NASA Marshall Space Flight Center.
WBS Title
SOS
1
LAUNCH
Factor-based approach - derived from
real data (c. Smart, SAIC)
WBS
Element
4.3
MISSION/GROUND OPS
Estimate Example
– NASA
WBS
Number
Upper Stage Avioncis
Software
Upper Stage Avionics
Hardware
Upper Stage Integration,
Assy., and C/O
Upper Stage SE&I
Upper Stage PM
Upper Stage STO
Upper Stage GSE
Mission/Ground Ops
Ground Stations
Launch Head
Wallops Island, VA
New Boston, NH
Argentia, Newfoundland
Deep Space Network (DSN)
Stations
Range Safety
CM Prime Mission Product
(HW + SW)
Systems Engineering,
Integration and Test and Top
Level Program Management
of SM
SM Prime Mission Product
(HW + SW)
additional satellite(s) to
support CxP ISS mission
SRM
Improvements for
communicationg with ground
and TDRSS.
Comms to/from AF ROCC
Provides -S-band and UHF
air-to- ground
MILA/PDL = Merritt Island &
Ponce de Leon; Merritt has
14 antennae; Ponce de Leon
has 3 antennae; both are
dedicated to shuttle/CxP
missions.
5m and 8m S-Band tracking;
not dedicated site
Canberra, Goldstone, Madrid
Backuo provided by DoD
ground-based radars
9
Recommendations
 Carefully consider the application
 Consider how SoS differs from traditional engineering systems and how
this affects the estimator
 Supporting platforms are operationally, geographic, and managerial independent,
as well as network-centric
 New acquisition concepts means we need new CERs, factors, and cost drivers
 Immaturity of concept means little cost data is currently available
 SOS cost drivers are unique and require considerations beyond
traditional systems estimating
 Review available research and papers
 Current research by USC, DAU, SEI, NASA, Cranfield University (UK)
 Available papers from USC, MIT, IEEE, INCOSE
10
Lessons Learned
 Inconsistent parameter definitions
 Between models (need to adopt INCOSE standards and develop a mapping scheme)
 Between historic data (collected before standards were established) and new CERs
 Inconsistent WBS applications
 Are we double-counting the SOS costs by applying factors at each platform level?
 Inconsistent platform applications
 Do we use the same factors for hardware systems as well as software systems? Are
space platform factors different from air and ground platform factors? Are mannedspace platform factors different from unmanned-space platform factors?
 Might be more useful if, in the near term, we rely on
databases (and factors) rather than on statistical CERs
11
Publication Schedule
 Highly motivated, volunteer workforce
 Hosted in a collaborative work area
 Periodic “tag-up” telecons
 Current Schedule
 Review 1 – 31 October 2010
 Review 2 – 20 November 2010
 Materials to Editor – 30 November 2010
 Complete DR first draft – 31 December 2010
 Seeking volunteer reviewers
 Published Desk Reference will be available in April 2011!
12