Jet Propulsion Laboratory
The Conundrums of the Costing World
Jairus Hihn
October 28, 2008
Phoenix
SMAP
23rd International Forum on
COCOMO and Systems/Software Cost Modeling
The research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology,
under a contract with the National Aeronautics and Space Administration
Background
The Jet Propulsion Laboratory is a Federally Funded
Research & Development Center operated by the California
Institute of Technology for the National Aeronautics and
Space Administration.
As part of the NASA team, JPL enables the nation to explore
space for the benefit of humankind by developing robotic
space missions to:
 Explore our own and neighboring planetary systems.
 Search for life beyond the Earth's confines.
 Further our understanding of the origins and evolution of the universe
and the laws that govern it.
 Enable a virtual presence throughout the solar system using the Deep
Space Network and evolving it to the Interplanetary Network of the
future.
2
Measurement Estimation & Analysis
“Creating a Quantitative SW Management Culture”
Helping Projects
Measuring Development Performance
Size Growth
& Stability
Resources
& Cost
• SW Cost Estimation Handbook
• Software Cost Estimation Class
• Estimation Support
• Estimation Tools
– Flight SW Cost Model (Team X)
– SCAT: Probabilistic COCOMO
– SLIC: Code Counter
Schedule &
Progress
Product
Quality
• Project SW Measures Guide
• Managing using Software Metrics Class
• Measurement Support
• Software Repositories
– Cost
– Defects
– Foundation Measures
• SW Engineering Models to support task planning
Source Line s Of Code (SLOC) De v e lopme nt Progre ss
30000
Total Estimated SLOCs
Code delivery to formal
softw are test11/20/02
Actual SLOCs
Projected SLOCs
25000 SLOC development threshold limit
20000
SLOCs
Estimation
& Planning
16517
15000
10000
Softw are integration on the
Engineering Unit DM
5000
0
Apr-01
Jun-01 Aug-01
Oct-01
Dec-01 Feb-02 Apr-02
Jun-02 Aug-02
Oct-02
Dec-02 Feb-03 Apr-03
Jun-03
Organizational and Process Measures
• JPL SW baselines and trends • Measure Impact of SQI
– Cost & Schedule Growth • Process Metrics
– Productivity
• Benchmarking
– Reuse
– Defect Density
– Cycle Time
Hihn
MEsA
1-3
Background (cont.)
JPL has around 5,000 employees
  800 work with software in some capacity
SQI has  15 FTE
MEsA  4 FTE
Assessed at Maturity Level 3 in September 2007
 Scope was Class B and C Mission Software
4
Cost Estimation Issues: Point 1
 All the people with power know just enough
to be dangerous
 Cost Estimators and Modelers must become
change agents
5
Models Work But Getting Buy-in is Still a Struggle
Jet Propulsion Laboratory
The research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology,
under a contract with the National Aeronautics and Space Administration
OCM Key Points
 Organizational Change does not come quickly or easily
 The improvement process needs to be approached with
many of the same deliberate methods and practices that
are used in actual system development.
 There are similarities between change at the individual
and organizational level
 The onus is on us to proactively reach out to
customer/others and promulgate change
 It is essential to proactively reach out to customers instead of
merely waiting for them to come to you.
7
The Onus Is On Us
 Think about it, what are the alternatives?
 Blaming others
won’t
they
tool/methodology?
 Why
(colleagues)
use
our
 Why don’t those #$@@ managers force them to
adopt our new approach?
 Repeatedly OCM studies have documented that this
approach never works
 The Change Agent must accept responsibility for
proactively reaching
promulgating change
out
to
‘customers’
and
8
Percentage in Each Adopter Category
Excerpted from http://www.valuebasedmanagement.net
9
OCM Stages
We Are Here
Internalization
COMMITMENT TO CHANGE
Institutionalization
Adoption
Initial Use
Understanding
Awareness
Contact
2
4
5
6
7
Mechanisms to support
sustaining the change
Mechanisms to support
wider rollout of change
Mechanisms to support
measured success in piloting
Mechanisms to assure understanding
3
Mechanisms to promote awareness
1
TIME
Hihn
Adapted from Out from Dependency: Thriving as an Insurgent in a Sometimes Hostile
Environment,
SuZ Garcia and Chuck Myers, SEPG Conference, 2001
1-10
Parametric Cost Estimation Evolution
Year
SW Cost Activities
Organizational
Reaction
OCM
19861989
Softcost: developed by Tauseworth and Reifer for DSN but
not used much
Collecting COCOMO metrics for ISS,
SCT: probabilistic COCOMO
JPL had very limited interest
and did not want to be
bothered.
Contact
19891991
JPL’s first SPI (SORCE) provided funding to collect
COCOMO data with effort borken out and detailed write ups
Focus on non-NASA projects,
later 2 GDS were willing to
participate.
Awareness
19901995
DSN Implementation Office paid us verify cost using SCT
and identify risks, collect metrics, 10 page reports.
Begrudging recognition was a
good thing
Initial Use
19962000
No or little institutional funding, isolated special requests for
some larger GDS projects.
Cost Growth Study
Two of the Mgrs we worked
with used models for main
estimates
Limited Use
Awareness in
Program Dir.
20012007
SDR requires multiples estimates
All day cost and metrics class
SCAT is advertised as one of SQI’s best products
Team X: Flight SW Cost Model and Chair
PSSE position is required
More and more managers and
reviewers expect to see model
based probabilistic estimates.
Projects often requested to
explain differences.
Adoption to
Institutionalization
Today
Growing number of very vocal supporters and independent
users of SCAT, especially in Flight SW community
SDSPs being revised and include direct references as a
requirement for model based estimates
Costing is ahead of rest of SQI
and we have mostly engaged
early majority.
Now facing late majority and
laggards.
Hints of
Internalization in early
adopters, but they are
changing!
Early Majority at Initial
Use
11
Cost Estimation Issues: Point 2
 Budget ‘bogies’ get set very early in lifecycle.
Sometimes based on casual conversations.
You will typically get held to this number!!
 Current proposal and planning process encourages/
demands under-estimating in early stages of lifecycle
12
What Should One Cost?
Development Cost
What Flies
Proposed
Performance Index
In our environment we need to know what it will cost
after all the *X!*! happens
13
Cost Estimation Issues: Point 3
 We improve our models but rarely do we improve our
techniques
 Our models have too many inputs
 If samples are small not only is the data not normal it is
not log normal either
14
Key Research Results Part 1
• Following is based on research with Tim Menzies
using data mining techniques to analyze COCOMO
• 2004-2008
• The results are …
• Our models have too many inputs
– Measures of RE go up with over specified models
– Problem is what is best varies
• Manual stratification does not lead to the ‘best’
model
– E.g. a combination of flight SW and Class B ground
produces a ‘better’ model then just selecting all
your flight records and doing LC.
– Nearest Neighbor searches for analogous records
based on your current project model inputs
15
Key Research Results Part 2
The results are …
• The same approach is
never best but some
combination of the
following always wins
– Local Calibration
– Column Pruning
– Row Pruning based on
Nearest Neighbor
• Which is best is
determined case-by-case
• Recalibrating every time
you estimate
16
Some Codicil Thoughts
• There is no such thing as a point estimate that means
anything, must account for uncertainty
• We built SCAT a probabilistic COCOMO in Excel
• Often the earliest simplest estimate is the best
• Too often we talk ourselves into unrealistic low
costs or get beaten into it.
• Give Me Multiple high level estimates any day
– Detailed estimates get lost in the trees and in an
environment of requirements volatility it is a house
of cards.
• Whatever you do be consistent
17